Sep 18 18:42:22.104207 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 18:42:22.104219 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 18:42:22.104228 ip-10-0-132-168 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 18:42:22.104425 ip-10-0-132-168 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 18:42:32.128165 ip-10-0-132-168 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 18:42:32.128186 ip-10-0-132-168 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b0f1149966a24972aab3f07f72fc77cc --
Sep 18 18:44:34.787140 ip-10-0-132-168 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 18:44:38.136600 ip-10-0-132-168 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:44:38.136600 ip-10-0-132-168 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 18:44:38.136600 ip-10-0-132-168 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:44:38.136600 ip-10-0-132-168 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 18:44:38.136600 ip-10-0-132-168 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 18:44:38.137120 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.136667    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 18:44:38.140175 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140159    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:44:38.140175 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140175    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140179    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140182    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140185    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140188    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140191    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140193    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140196    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140198    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140208    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140210    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140213    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140215    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140218    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140220    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140223    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140225    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140228    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140230    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140233    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:44:38.140239 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140236    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140240    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140243    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140246    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140250    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140253    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140256    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140259    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140261    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140264    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140266    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140268    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140270    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140273    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140276    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140278    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140281    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140283    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140285    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:44:38.140699 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140288    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140290    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140293    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140295    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140298    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140300    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140303    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140305    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140307    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140309    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140312    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140314    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140316    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140319    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140321    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140324    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140327    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140329    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140332    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140334    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:44:38.141163 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140336    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140338    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140341    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140343    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140345    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140348    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140350    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140353    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140355    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140357    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140360    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140362    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140365    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140367    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140370    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140373    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140376    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140378    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140380    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140384    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:44:38.141629 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140387    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140389    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140392    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140394    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140396    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140399    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140401    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140404    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140408    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140411    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140414    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140417    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140419    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140422    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140425    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140427    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140430    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140433    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.140436    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141104    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:44:38.142128 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141111    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141114    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141117    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141121    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141124    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141126    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141129    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141132    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141135    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141138    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141141    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141143    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141146    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141148    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141151    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141153    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141156    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141158    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141161    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141165    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:44:38.142593 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141168    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141171    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141173    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141176    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141179    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141181    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141184    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141186    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141189    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141192    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141194    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141197    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141199    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141202    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141204    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141207    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141209    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141212    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141215    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:44:38.143103 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141218    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141221    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141228    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141231    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141234    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141236    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141238    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141241    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141243    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141245    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141248    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141250    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141253    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141256    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141258    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141261    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141263    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141266    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141268    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141270    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:44:38.143552 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141273    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141275    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141277    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141280    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141282    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141285    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141287    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141289    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141292    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141294    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141296    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141298    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141301    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141303    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141307    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141315    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141318    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141321    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141323    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:44:38.144041 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141326    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141330    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141333    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141336    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141339    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141342    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141345    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141348    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141350    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141352    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141355    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141357    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141360    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141362    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141365    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141367    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141369    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141372    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141374    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:44:38.144520 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141377    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141771    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141776    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141779    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141782    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141785    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141788    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141790    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141793    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141796    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141798    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141801    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141814    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141818    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141820    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141823    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141826    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141829    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141832    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141835    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:44:38.144992 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141838    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141840    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141843    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141845    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141848    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141850    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141853    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141855    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141858    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141860    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141862    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141865    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141869    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141871    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141873    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141876    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141896    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141900    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141904    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141907    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:44:38.145464 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141909    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141912    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141915    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141917    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141920    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141922    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141925    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141927    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141930    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141932    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141937    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141939    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141942    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141944    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141947    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141949    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141953    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141956    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141959    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:44:38.145996 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141962    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141964    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141967    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141970    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141972    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141975    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141977    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141979    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141982    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141984    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141989    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141993    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141995    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.141998    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142000    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142002    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142007    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142010    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142012    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:44:38.146442 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142017    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142021    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142024    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142027    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142031    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142033    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142036    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142038    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142040    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142043    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142045    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142048    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142050    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142052    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142054    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142057    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142059    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142061    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142064    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142066    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:44:38.146910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142069    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142071    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142131    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142162    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142169    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142173    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142177    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142180    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142184    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142187    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142191    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142193    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142197    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142200    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142203    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142206    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142209    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142212    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142215    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142218    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142221    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142224    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142227    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142231    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 18:44:38.147409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142234    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142237    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142240    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142243    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142246    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142249    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142252    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142254    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142257    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142260    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142264    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142267    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142271    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142274    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142277    2582 flags.go:64] FLAG: --enable-server="true"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142279    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142283    2582 flags.go:64] FLAG: --event-burst="100"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142286    2582 flags.go:64] FLAG: --event-qps="50"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142289    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142292    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142295    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142298    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142301    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142303    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142307    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 18 18:44:38.147981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142310    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142313    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142315    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142318    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142321    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142323    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142326    2582 flags.go:64] FLAG: --feature-gates=""
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142329    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142332    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142335    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142337    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142340    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142343    2582 flags.go:64] FLAG: --help="false"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142346    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142348    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142351    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142354    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142357    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142360    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142365    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142367    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142370    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142373    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 18:44:38.148578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142375    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142378    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142381    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142383    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142386    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142389    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142392    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142395    2582 flags.go:64] FLAG: --lock-file=""
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142397    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142400    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142403    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142410    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142413    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142415    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142418    2582 flags.go:64] FLAG: --logging-format="text"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142421    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142424    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142426    2582 flags.go:64] FLAG: --manifest-url=""
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142429    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142433    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142436    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142440    2582 flags.go:64] FLAG: --max-pods="110"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142442    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142445    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142448    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 18:44:38.149624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142451    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142453    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142456    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142460    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142466    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142469    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142472    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142474    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142477    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142482    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142484    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142488    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142491    2582 flags.go:64] FLAG: --port="10250"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142494    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142497    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09233febd62553d83"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142500    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142502    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142505    2582 flags.go:64] FLAG: --register-node="true"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142508    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142511    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142514    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142516    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142519    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142522    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142525    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 18:44:38.150659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142528    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142531    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142533    2582 flags.go:64] FLAG: --runonce="false"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142536    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142539    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142542    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142544    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142548    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142551    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142554    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142557    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142561    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142564    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142566    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142569    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142572    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142575    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142577    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142583    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142585    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142589    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142592    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142595    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142597    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142600    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 18:44:38.151577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142603    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142606    2582 flags.go:64] FLAG: --v="2"
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142609    2582 flags.go:64] FLAG: --version="false"
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142613    2582 flags.go:64] FLAG: --vmodule=""
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142616    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.142619    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142732    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142736    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142739    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142742    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142744    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142747    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142750    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142752    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142756    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142758    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142761    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142764    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142767    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142770    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142772    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:44:38.152194 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142775    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142777    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142780    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142782    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142785    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142787    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142790    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142793    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142795    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142798    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142800    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142803    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142805    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142808    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142810    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142813    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142815    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142817    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142820    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142822    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:44:38.152689 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142825    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142827    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142830    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142832    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142834    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142840    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142843    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142846    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142849    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142853    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142857    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142860    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142863    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142865    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142868    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142870    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142873    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142875    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142893    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:44:38.153171 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142897    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142901    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142904    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142907    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142909    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142912    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142916    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142919    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142922    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142925    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142927    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142930    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142932    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142934    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142937    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142939    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142942    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142944    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142948    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:44:38.153624 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142950    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142953    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142956    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142960    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142962    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142965    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142967    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142970    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142972    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142975    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142977    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142980    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142982    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142985    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142987    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142990    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142992    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142995    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.142997    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.143000    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:44:38.154097 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.143002    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.143005    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.143007    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.143009    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.143012    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.143014    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.143767    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.151604    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.151617    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.151637    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.151645    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151703    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151708    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151711    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151714    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:44:38.154565 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151716    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151719    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151722    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151725    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151727    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151730    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151732    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151735    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151737    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151740    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151743    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151745    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151748    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151750    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151753    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151755    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151758    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151760    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151763    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151765    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:44:38.155081 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151768    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151771    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151773    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151776    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151778    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151782    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151784    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151787    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151789    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151792    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151794    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151797    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151799    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151802    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151804    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151806    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151809    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151811    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151814    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151817    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:44:38.155553 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151819    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151822    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151824    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151827    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151829    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151832    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151834    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151836    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151839    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151841    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151844    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151846    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151849    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151851    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151854    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151856    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151859    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151862    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151864    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151866    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:44:38.156042 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151869    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151871    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151874    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151877    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151896    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151899    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151902    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151905    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151908    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151911    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151913    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151915    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151918    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151920    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151923    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151926    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151928    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151931    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151934    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:44:38.156551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151936    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151938    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151941    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151944    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151947    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151950    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151953    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151956    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151958    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151961    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151964    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151967    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151969    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151972    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151975    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:44:38.157018 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.151979    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.151984    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152086    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152091    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152094    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152097    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152099    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152102    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152105    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152108    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152110    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152113    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152115    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152118    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152122    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 18:44:38.157392 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152124    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152127    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152129    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152132    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152134    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152136    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152139    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152141    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152144    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152146    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152149    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152151    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152154    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152156    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152159    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152162    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152167    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152169    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152172    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152175    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 18:44:38.157761 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152177    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152180    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152182    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152185    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152187    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152190    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152192    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152195    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152197    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152200    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152203    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152205    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152208    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152211    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152213    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152215    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152218    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152220    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152223    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 18:44:38.158286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152225    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152228    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152230    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152233    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152235    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152238    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152242    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152246    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152249    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152251    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152254    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152256    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152259    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152261    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152264    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152266    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152268    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152271    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152273    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 18:44:38.158767 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152276    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152279    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152282    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152285    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152288    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152291    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152294    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152297    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152300    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152302    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152305    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152307    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152310    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152312    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152315    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152317    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152320    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152322    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152324    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 18:44:38.163324 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152327    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152330    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152332    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152334    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152337    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152339    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152342    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152344    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:38.152347    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.152351    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.152449    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.154459    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.155194    2582 server.go:1019] "Starting client certificate rotation"
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.155286    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 18:44:38.163813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.155316    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 18:44:38.164157 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.157425    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 18:44:38.164157 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.158519    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 18:44:38.164949 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.164933    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 18 18:44:38.166336 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.166322    2582 log.go:25] "Validated CRI v1 image API"
Sep 18 18:44:38.167095 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.167079    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 18:44:38.171008 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.170990    2582 fs.go:135] Filesystem UUIDs: map[35fa0208-f513-4f75-869e-6170669df946:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f2ab16ac-8ddc-4be0-a5f0-a0ff395ec503:/dev/nvme0n1p4]
Sep 18 18:44:38.171068 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.171007    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 18:44:38.176349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.176250    2582 manager.go:216] Machine: {Timestamp:2026-09-18 18:44:38.17499251 +0000 UTC m=+1.967669442 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3173076 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24548cadb34d7884ccac63d1dcea85 SystemUUID:ec24548c-adb3-4d78-84cc-ac63d1dcea85 BootID:b0f11499-66a2-4972-aab3-f07f72fc77cc Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:b1:95:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:b1:95:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:47:81:5b:a0:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 18:44:38.176349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.176348    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 18:44:38.176450 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.176431    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 18:44:38.177833 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.177808    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 18:44:38.178028 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.177838    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 18:44:38.179020 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.179011    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 18:44:38.179055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.179023    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 18:44:38.179055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.179031    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 18:44:38.179055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.179043    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 18:44:38.180148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.180138    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 18:44:38.182155 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.182141    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 18:44:38.183996 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.183985    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 18:44:38.184114 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184105    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 18:44:38.184148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184141    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 18:44:38.184182 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184169    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 18:44:38.184221 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184186    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 18:44:38.184221 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184195    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 18 18:44:38.184221 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184203    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 18:44:38.184566 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184551    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 18:44:38.184627 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.184569    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 18:44:38.195831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.195808    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 18:44:38.197755 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.197741    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 18:44:38.197806 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.197800    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 18:44:38.198762 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198751    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 18:44:38.198798 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198767    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 18:44:38.198798 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198773    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 18:44:38.198798 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198778    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 18:44:38.198798 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198786    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 18:44:38.198798 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198791    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198803    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198808    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198815    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198819    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198827    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198832    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198856    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 18:44:38.198942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.198860    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 18:44:38.200731 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.200687    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 18:44:38.200793 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.200687    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 18:44:38.202258 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.202244    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 18:44:38.202807 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.202791    2582 server.go:1268] "Started kubelet"
Sep 18 18:44:38.202902 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.202858    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 18:44:38.202953 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.202895    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 18:44:38.202997 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.202960    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 18:44:38.203575 ip-10-0-132-168 systemd[1]: Started Kubernetes Kubelet.
Sep 18 18:44:38.204118 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.203867    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 18:44:38.204118 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.204089    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 18:44:38.205694 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.205677    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8fjq7"
Sep 18 18:44:38.206420 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.206403    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 18:44:38.212872 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.212857    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 18:44:38.212960 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.212875    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 18:44:38.212960 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.212944    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.213419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.213405    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8fjq7"
Sep 18 18:44:38.213483 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.213435    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 18:44:38.213483 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.213449    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 18:44:38.213483 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.213437    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 18:44:38.213606 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.213504    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 18:44:38.213606 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.213512    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 18:44:38.214717 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.214697    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 18:44:38.214824 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.214790    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 18:44:38.214909 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.214899    2582 factory.go:55] Registering systemd factory
Sep 18 18:44:38.214971 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.214913    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 18 18:44:38.215153 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.215136    2582 factory.go:153] Registering CRI-O factory
Sep 18 18:44:38.215240 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.215155    2582 factory.go:223] Registration of the crio container factory successfully
Sep 18 18:44:38.215240 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.215212    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 18:44:38.215240 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.215240    2582 factory.go:103] Registering Raw factory
Sep 18 18:44:38.215375 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.215256    2582 manager.go:1196] Started watching for new ooms in manager
Sep 18 18:44:38.215862 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.215850    2582 manager.go:349] Starting recovery of all containers
Sep 18 18:44:38.237423 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.237407    2582 manager.go:354] Recovery completed
Sep 18 18:44:38.241619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.241602    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:44:38.243148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.243133    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:44:38.243238 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.243165    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:44:38.243238 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.243180    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:44:38.243710 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.243692    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 18:44:38.243710 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.243706    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 18:44:38.243849 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.243724    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 18:44:38.243849 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.243725    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:44:38.244431 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.244413    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:44:38.244508 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.244444    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:44:38.244508 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.244457    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:44:38.245731 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.245717    2582 policy_none.go:49] "None policy: Start"
Sep 18 18:44:38.245811 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.245737    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 18:44:38.245811 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.245750    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 18:44:38.247772 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.247759    2582 policy_none.go:47] "Start"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.285404    2582 manager.go:337] "Starting Device Plugin manager"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.285565    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.285575    2582 server.go:87] "Starting device plugin registration server"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.285745    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.285761    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.285905    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.286018    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.286027    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.286361    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 18:44:38.289768 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.286409    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.327206 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.327189    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 18:44:38.328440 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.328421    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 18:44:38.328517 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.328461    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 18:44:38.328517 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.328480    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 18:44:38.328616 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.328520    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 18:44:38.330774 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.330759    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:44:38.386838 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.386791    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:44:38.387534 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.387518    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:44:38.387609 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.387550    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:44:38.387609 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.387565    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:44:38.387609 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.387591    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.395289 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.395276    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.395329 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.395295    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-168.ec2.internal\": node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.407036 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.407016    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.429009 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.428990    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"]
Sep 18 18:44:38.429056 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.429041    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:44:38.429737 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.429723    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:44:38.429796 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.429749    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:44:38.429796 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.429763    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:44:38.431999 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.431988    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:44:38.432147 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.432179 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432161    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:44:38.432630 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432607    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:44:38.432752 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432635    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:44:38.432752 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432649    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:44:38.432752 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432668    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:44:38.432752 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432687    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:44:38.432752 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.432701    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:44:38.434959 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.434944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.435034 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.434967    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 18:44:38.435571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.435555    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 18:44:38.435637 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.435578    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 18:44:38.435637 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.435591    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeHasSufficientPID"
Sep 18 18:44:38.448381 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.448364    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-168.ec2.internal\" not found" node="ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.452206 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.452190    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-168.ec2.internal\" not found" node="ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.508102 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.508083    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.515808 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.515786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/911c34c29a66f4fca404d6224fced36c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"911c34c29a66f4fca404d6224fced36c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.515863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.515814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/911c34c29a66f4fca404d6224fced36c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"911c34c29a66f4fca404d6224fced36c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.515863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.515829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aba6b42d05a06c053c050c26af669cbb-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"aba6b42d05a06c053c050c26af669cbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.608981 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.608963    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.616321 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.616300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/911c34c29a66f4fca404d6224fced36c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"911c34c29a66f4fca404d6224fced36c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.616380 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.616326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/911c34c29a66f4fca404d6224fced36c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"911c34c29a66f4fca404d6224fced36c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.616380 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.616341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aba6b42d05a06c053c050c26af669cbb-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"aba6b42d05a06c053c050c26af669cbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.616380 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.616369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aba6b42d05a06c053c050c26af669cbb-config\") pod \"kube-apiserver-proxy-ip-10-0-132-168.ec2.internal\" (UID: \"aba6b42d05a06c053c050c26af669cbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.616481 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.616383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/911c34c29a66f4fca404d6224fced36c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"911c34c29a66f4fca404d6224fced36c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.616481 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.616402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/911c34c29a66f4fca404d6224fced36c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal\" (UID: \"911c34c29a66f4fca404d6224fced36c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.709728 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.709686    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.752115 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.752099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.755601 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:38.755590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:38.810443 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.810420    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:38.910926 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:38.910907    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:39.011402 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.011354    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:39.111793 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.111772    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-168.ec2.internal\" not found"
Sep 18 18:44:39.153205 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.153184    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:44:39.155236 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.155213    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 18:44:39.155386 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.155305    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 18:44:39.155386 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.155308    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 18:44:39.185599 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.185576    2582 apiserver.go:52] "Watching apiserver"
Sep 18 18:44:39.199748 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.199729    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 18:44:39.200096 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.200073    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-78vcs","openshift-multus/multus-85kwk","openshift-multus/multus-additional-cni-plugins-vszhp","openshift-multus/network-metrics-daemon-6xbrr","openshift-network-diagnostics/network-check-target-ghpld","kube-system/global-pull-secret-syncer-hrxqr","kube-system/konnectivity-agent-7xv6g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg","openshift-cluster-node-tuning-operator/tuned-klds4","openshift-network-operator/iptables-alerter-hxcfx","openshift-ovn-kubernetes/ovnkube-node-5kmhl","openshift-dns/node-resolver-hf5v7"]
Sep 18 18:44:39.202527 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.202505    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.204620 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.204584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.204904 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.204870    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ctjn9\""
Sep 18 18:44:39.205146 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.205128    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.205146 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.205139    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 18:44:39.205286 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.205157    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.206321 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.206300    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 18:44:39.207192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.206829    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 18:44:39.207192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.207082    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.207192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.207165    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.207369 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.207308    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wfw6r\""
Sep 18 18:44:39.207672 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.207654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.209687 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.209667    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:39.209781 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.209750    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 18:44:39.209839 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.209749    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:39.209839 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.209750    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 18:44:39.209839 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.209825    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2gf9\""
Sep 18 18:44:39.211743 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.211728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:39.211826 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.211779    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:39.212974 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.212958    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 18:44:39.213073 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.213059    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:39.213768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.213752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.213859 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.213803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:39.215976 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.215963    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.217861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.217844    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-68xzp\""
Sep 18 18:44:39.217976 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.217960    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 18:44:39.218042 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.217990    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 18:44:39.218163 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.218146    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.219117 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:39.219200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-cni-multus\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-etc-kubernetes\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0f8445-381e-4d78-b730-ac9eb826b144-dbus\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.219200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b2eac25-004e-49a9-ba24-15fe12ebfe13-host\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-netns\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7t69\" (UniqueName: \"kubernetes.io/projected/9b2eac25-004e-49a9-ba24-15fe12ebfe13-kube-api-access-h7t69\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-cni-bin\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-system-cni-dir\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vld\" (UniqueName: \"kubernetes.io/projected/129a6d33-4815-4918-9f85-bbe434e773ea-kube-api-access-g7vld\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/577500df-cb0a-418e-a573-688a25c63dbb-agent-certs\") pod \"konnectivity-agent-7xv6g\" (UID: \"577500df-cb0a-418e-a573-688a25c63dbb\") " pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cnibin\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.219563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-cnibin\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-socket-dir-parent\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-k8s-cni-cncf-io\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-kubelet\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-hostroot\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b2eac25-004e-49a9-ba24-15fe12ebfe13-serviceca\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219797    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-system-cni-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.219873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-cni-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-conf-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.219934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-multus-certs\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zr84\" (UniqueName: \"kubernetes.io/projected/c767a1b9-30ea-4bff-801a-59c85191b563-kube-api-access-6zr84\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220040    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c767a1b9-30ea-4bff-801a-59c85191b563-multus-daemon-config\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.220148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220213    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jrxx7\""
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/577500df-cb0a-418e-a573-688a25c63dbb-konnectivity-ca\") pod \"konnectivity-agent-7xv6g\" (UID: \"577500df-cb0a-418e-a573-688a25c63dbb\") " pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0f8445-381e-4d78-b730-ac9eb826b144-kubelet-config\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zclv\" (UniqueName: \"kubernetes.io/projected/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-kube-api-access-7zclv\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.220464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.220392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c767a1b9-30ea-4bff-801a-59c85191b563-cni-binary-copy\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.222363 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.222348    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.222494 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.222480    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vp66d\""
Sep 18 18:44:39.222783 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.222770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.222937 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.222919    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.223811 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.223772    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 18:39:38 +0000 UTC" deadline="2028-04-08 23:28:06.151532157 +0000 UTC"
Sep 18 18:44:39.223811 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.223810    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13636h43m26.92772429s"
Sep 18 18:44:39.225207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.224829    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.225207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.224856    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 18:44:39.225207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.224953    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"
Sep 18 18:44:39.225207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.225096    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.225207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.225133    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 18:44:39.225651 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.225636    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wfggf\""
Sep 18 18:44:39.225765 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.225751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.227659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.227645    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 18:44:39.227798 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.227788    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 18:44:39.228010 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.227996    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal"]
Sep 18 18:44:39.228065 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.228056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.229143 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.229126    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 18:44:39.229143 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.229137    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 18:44:39.229278 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.229150    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 18:44:39.229278 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.229151    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.229278 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.229178    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.229278 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.229217    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bgjfn\""
Sep 18 18:44:39.229977 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.229960    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 18:44:39.230088 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.230075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dj59h\""
Sep 18 18:44:39.230138 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.230081    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 18:44:39.233166 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.233152    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 18:44:39.233377 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.233363    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal"]
Sep 18 18:44:39.253683 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.253658    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gcxdx"
Sep 18 18:44:39.260680 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.260607    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gcxdx"
Sep 18 18:44:39.314013 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.313996    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 18:44:39.320828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.320808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.320912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.320844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37cf107-9c81-4955-8fd0-a54129f38f93-host-slash\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.320912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.320869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-slash\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.320912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.320906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbl7k\" (UniqueName: \"kubernetes.io/projected/aba42289-c2a5-4752-87a8-f616b4372de0-kube-api-access-fbl7k\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.321016 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.320933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-cni-multus\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.321016 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.320982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-sys-fs\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.321016 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-etc-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.321122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b2eac25-004e-49a9-ba24-15fe12ebfe13-host\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.321122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-cni-multus\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.321122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysctl-conf\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.321122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d9edbb-99b9-44d1-81ef-9e609393c40b-tmp\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.321122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b2eac25-004e-49a9-ba24-15fe12ebfe13-host\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.321122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsc8z\" (UniqueName: \"kubernetes.io/projected/d3d9edbb-99b9-44d1-81ef-9e609393c40b-kube-api-access-fsc8z\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.321122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-env-overrides\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkl6\" (UniqueName: \"kubernetes.io/projected/99b431ce-f544-49f6-bc43-38ae2c257ae6-kube-api-access-qqkl6\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0f8445-381e-4d78-b730-ac9eb826b144-kubelet-config\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t69\" (UniqueName: \"kubernetes.io/projected/9b2eac25-004e-49a9-ba24-15fe12ebfe13-kube-api-access-h7t69\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-system-cni-dir\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vld\" (UniqueName: \"kubernetes.io/projected/129a6d33-4815-4918-9f85-bbe434e773ea-kube-api-access-g7vld\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:39.321349 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d0f8445-381e-4d78-b730-ac9eb826b144-kubelet-config\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-system-cni-dir\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-systemd-units\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c767a1b9-30ea-4bff-801a-59c85191b563-multus-daemon-config\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-tuned\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cnibin\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-cnibin\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.321604    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:39.321610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-socket-dir-parent\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cnibin\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b2eac25-004e-49a9-ba24-15fe12ebfe13-serviceca\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-system-cni-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-cnibin\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.321671    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret podName:4d0f8445-381e-4d78-b730-ac9eb826b144 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:39.821649048 +0000 UTC m=+3.614325959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret") pod "global-pull-secret-syncer-hrxqr" (UID: "4d0f8445-381e-4d78-b730-ac9eb826b144") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-system-cni-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-conf-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-modprobe-d\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-k8s-cni-cncf-io\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-socket-dir-parent\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-hostroot\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-k8s-cni-cncf-io\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-kubelet\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321913    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-hostroot\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321924    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-conf-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.321965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-systemd\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c767a1b9-30ea-4bff-801a-59c85191b563-multus-daemon-config\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovnkube-script-lib\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-registration-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnq6\" (UniqueName: \"kubernetes.io/projected/d6697d2f-f8b5-400f-8b9b-a59f037557a2-kube-api-access-gnnq6\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b2eac25-004e-49a9-ba24-15fe12ebfe13-serviceca\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-sys\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk9p\" (UniqueName: \"kubernetes.io/projected/e37cf107-9c81-4955-8fd0-a54129f38f93-kube-api-access-brk9p\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-run-netns\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-systemd\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovnkube-config\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c767a1b9-30ea-4bff-801a-59c85191b563-cni-binary-copy\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:39.322778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-lib-modules\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-host\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37cf107-9c81-4955-8fd0-a54129f38f93-iptables-alerter-script\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322427    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-node-log\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba42289-c2a5-4752-87a8-f616b4372de0-tmp-dir\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.322463    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-etc-kubernetes\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0f8445-381e-4d78-b730-ac9eb826b144-dbus\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-etc-kubernetes\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.322573    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs podName:129a6d33-4815-4918-9f85-bbe434e773ea nodeName:}" failed. No retries permitted until 2026-09-18 18:44:39.822556558 +0000 UTC m=+3.615233479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs") pod "network-metrics-daemon-6xbrr" (UID: "129a6d33-4815-4918-9f85-bbe434e773ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-kubernetes\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-ovn\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aba42289-c2a5-4752-87a8-f616b4372de0-hosts-file\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322645    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d0f8445-381e-4d78-b730-ac9eb826b144-dbus\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-netns\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-cni-bin\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.323396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322685    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-netns\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-cni-netd\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysctl-d\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c767a1b9-30ea-4bff-801a-59c85191b563-cni-binary-copy\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-cni-bin\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/577500df-cb0a-418e-a573-688a25c63dbb-agent-certs\") pod \"konnectivity-agent-7xv6g\" (UID: \"577500df-cb0a-418e-a573-688a25c63dbb\") " pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zclv\" (UniqueName: \"kubernetes.io/projected/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-kube-api-access-7zclv\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-socket-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-cni-bin\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysconfig\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322940    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-var-lib-kubelet\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.322964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-kubelet\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovn-node-metrics-cert\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-cni-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-multus-certs\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-var-lib-kubelet\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zr84\" (UniqueName: \"kubernetes.io/projected/c767a1b9-30ea-4bff-801a-59c85191b563-kube-api-access-6zr84\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.323932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323112    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-multus-cni-dir\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c767a1b9-30ea-4bff-801a-59c85191b563-host-run-multus-certs\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-log-socket\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323164    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/577500df-cb0a-418e-a573-688a25c63dbb-konnectivity-ca\") pod \"konnectivity-agent-7xv6g\" (UID: \"577500df-cb0a-418e-a573-688a25c63dbb\") " pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-device-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-run\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-var-lib-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.324399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.323598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/577500df-cb0a-418e-a573-688a25c63dbb-konnectivity-ca\") pod \"konnectivity-agent-7xv6g\" (UID: \"577500df-cb0a-418e-a573-688a25c63dbb\") " pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.326053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.326038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/577500df-cb0a-418e-a573-688a25c63dbb-agent-certs\") pod \"konnectivity-agent-7xv6g\" (UID: \"577500df-cb0a-418e-a573-688a25c63dbb\") " pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.327537 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.327524    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:44:39.327573 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.327542    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:44:39.327573 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.327550    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mlvd5 for pod openshift-network-diagnostics/network-check-target-ghpld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:39.327630 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.327599    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5 podName:3fe33d9a-e11a-4bb4-8535-777727b79a84 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:39.827587836 +0000 UTC m=+3.620264748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mlvd5" (UniqueName: "kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5") pod "network-check-target-ghpld" (UID: "3fe33d9a-e11a-4bb4-8535-777727b79a84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:39.329849 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.329826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t69\" (UniqueName: \"kubernetes.io/projected/9b2eac25-004e-49a9-ba24-15fe12ebfe13-kube-api-access-h7t69\") pod \"node-ca-78vcs\" (UID: \"9b2eac25-004e-49a9-ba24-15fe12ebfe13\") " pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.332055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.332038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zclv\" (UniqueName: \"kubernetes.io/projected/a87e2cb4-40dc-4b7a-ba56-2d48085367f1-kube-api-access-7zclv\") pod \"multus-additional-cni-plugins-vszhp\" (UID: \"a87e2cb4-40dc-4b7a-ba56-2d48085367f1\") " pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.332294 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.332278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zr84\" (UniqueName: \"kubernetes.io/projected/c767a1b9-30ea-4bff-801a-59c85191b563-kube-api-access-6zr84\") pod \"multus-85kwk\" (UID: \"c767a1b9-30ea-4bff-801a-59c85191b563\") " pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.332412 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.332394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vld\" (UniqueName: \"kubernetes.io/projected/129a6d33-4815-4918-9f85-bbe434e773ea-kube-api-access-g7vld\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:39.388264 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.388238    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba6b42d05a06c053c050c26af669cbb.slice/crio-8cb1aeb887d2d737932bb8adbb7eef524a63c101d45319abd365161143fdf0dc WatchSource:0}: Error finding container 8cb1aeb887d2d737932bb8adbb7eef524a63c101d45319abd365161143fdf0dc: Status 404 returned error can't find the container with id 8cb1aeb887d2d737932bb8adbb7eef524a63c101d45319abd365161143fdf0dc
Sep 18 18:44:39.388661 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.388644    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911c34c29a66f4fca404d6224fced36c.slice/crio-445f2d41df5847bb2178ae3158b092eb46b806050eeeab3afbba1481a6fd5cc9 WatchSource:0}: Error finding container 445f2d41df5847bb2178ae3158b092eb46b806050eeeab3afbba1481a6fd5cc9: Status 404 returned error can't find the container with id 445f2d41df5847bb2178ae3158b092eb46b806050eeeab3afbba1481a6fd5cc9
Sep 18 18:44:39.393409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.393395    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:44:39.424608 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysctl-d\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.424703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-socket-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.424703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424647    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysconfig\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.424703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-var-lib-kubelet\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.424703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovn-node-metrics-cert\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-log-socket\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysctl-d\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysconfig\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-device-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-socket-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-var-lib-kubelet\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-log-socket\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-run\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-var-lib-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424909    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.424929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37cf107-9c81-4955-8fd0-a54129f38f93-host-slash\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-run\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-slash\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-var-lib-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbl7k\" (UniqueName: \"kubernetes.io/projected/aba42289-c2a5-4752-87a8-f616b4372de0-kube-api-access-fbl7k\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-device-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.424977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37cf107-9c81-4955-8fd0-a54129f38f93-host-slash\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-slash\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-sys-fs\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-etc-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysctl-conf\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d9edbb-99b9-44d1-81ef-9e609393c40b-tmp\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-sys-fs\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsc8z\" (UniqueName: \"kubernetes.io/projected/d3d9edbb-99b9-44d1-81ef-9e609393c40b-kube-api-access-fsc8z\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-etc-openvswitch\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-env-overrides\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.425571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkl6\" (UniqueName: \"kubernetes.io/projected/99b431ce-f544-49f6-bc43-38ae2c257ae6-kube-api-access-qqkl6\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-sysctl-conf\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-systemd-units\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-systemd-units\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-tuned\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-modprobe-d\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-kubelet\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-systemd\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovnkube-script-lib\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-registration-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425433    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-kubelet\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-env-overrides\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.426366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnq6\" (UniqueName: \"kubernetes.io/projected/d6697d2f-f8b5-400f-8b9b-a59f037557a2-kube-api-access-gnnq6\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-sys\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425716    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6697d2f-f8b5-400f-8b9b-a59f037557a2-registration-dir\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk9p\" (UniqueName: \"kubernetes.io/projected/e37cf107-9c81-4955-8fd0-a54129f38f93-kube-api-access-brk9p\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-run-netns\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-systemd\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.425840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovnkube-config\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-modprobe-d\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-run-netns\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-lib-modules\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-host\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37cf107-9c81-4955-8fd0-a54129f38f93-iptables-alerter-script\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-node-log\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba42289-c2a5-4752-87a8-f616b4372de0-tmp-dir\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-sys\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-kubernetes\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426198    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovnkube-script-lib\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-ovn\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-kubernetes\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-ovn\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aba42289-c2a5-4752-87a8-f616b4372de0-hosts-file\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-cni-bin\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426297    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aba42289-c2a5-4752-87a8-f616b4372de0-hosts-file\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-cni-netd\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-node-log\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-cni-netd\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-run-systemd\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99b431ce-f544-49f6-bc43-38ae2c257ae6-host-cni-bin\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovnkube-config\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426545    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-lib-modules\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-host\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-systemd\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba42289-c2a5-4752-87a8-f616b4372de0-tmp-dir\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.426744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37cf107-9c81-4955-8fd0-a54129f38f93-iptables-alerter-script\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.427166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99b431ce-f544-49f6-bc43-38ae2c257ae6-ovn-node-metrics-cert\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.427374    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3d9edbb-99b9-44d1-81ef-9e609393c40b-tmp\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.427522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.427445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3d9edbb-99b9-44d1-81ef-9e609393c40b-etc-tuned\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.437530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.437499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbl7k\" (UniqueName: \"kubernetes.io/projected/aba42289-c2a5-4752-87a8-f616b4372de0-kube-api-access-fbl7k\") pod \"node-resolver-hf5v7\" (UID: \"aba42289-c2a5-4752-87a8-f616b4372de0\") " pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.437972 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.437955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkl6\" (UniqueName: \"kubernetes.io/projected/99b431ce-f544-49f6-bc43-38ae2c257ae6-kube-api-access-qqkl6\") pod \"ovnkube-node-5kmhl\" (UID: \"99b431ce-f544-49f6-bc43-38ae2c257ae6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.442378 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.442358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsc8z\" (UniqueName: \"kubernetes.io/projected/d3d9edbb-99b9-44d1-81ef-9e609393c40b-kube-api-access-fsc8z\") pod \"tuned-klds4\" (UID: \"d3d9edbb-99b9-44d1-81ef-9e609393c40b\") " pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.460785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.460768    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk9p\" (UniqueName: \"kubernetes.io/projected/e37cf107-9c81-4955-8fd0-a54129f38f93-kube-api-access-brk9p\") pod \"iptables-alerter-hxcfx\" (UID: \"e37cf107-9c81-4955-8fd0-a54129f38f93\") " pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.463405 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.463387    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnq6\" (UniqueName: \"kubernetes.io/projected/d6697d2f-f8b5-400f-8b9b-a59f037557a2-kube-api-access-gnnq6\") pod \"aws-ebs-csi-driver-node-xktkg\" (UID: \"d6697d2f-f8b5-400f-8b9b-a59f037557a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.522272 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.522228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-78vcs"
Sep 18 18:44:39.531286 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.531268    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2eac25_004e_49a9_ba24_15fe12ebfe13.slice/crio-8ecee8d6d74302f3c6f4ffb352d118656612b2d06852abeda9bcd5e9f0bee41a WatchSource:0}: Error finding container 8ecee8d6d74302f3c6f4ffb352d118656612b2d06852abeda9bcd5e9f0bee41a: Status 404 returned error can't find the container with id 8ecee8d6d74302f3c6f4ffb352d118656612b2d06852abeda9bcd5e9f0bee41a
Sep 18 18:44:39.537269 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.537255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-85kwk"
Sep 18 18:44:39.544512 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.544495    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc767a1b9_30ea_4bff_801a_59c85191b563.slice/crio-3ac0b0e8c1d2202b0d81d831f8d386f0ebfec95197d6073ec8719bf91d40dabb WatchSource:0}: Error finding container 3ac0b0e8c1d2202b0d81d831f8d386f0ebfec95197d6073ec8719bf91d40dabb: Status 404 returned error can't find the container with id 3ac0b0e8c1d2202b0d81d831f8d386f0ebfec95197d6073ec8719bf91d40dabb
Sep 18 18:44:39.556516 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.556498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vszhp"
Sep 18 18:44:39.563287 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.563273    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:44:39.563646 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.563629    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87e2cb4_40dc_4b7a_ba56_2d48085367f1.slice/crio-4a3ddef44b69a85b50fa98765632ff3c13d64f12dbd077547785efc5015987ea WatchSource:0}: Error finding container 4a3ddef44b69a85b50fa98765632ff3c13d64f12dbd077547785efc5015987ea: Status 404 returned error can't find the container with id 4a3ddef44b69a85b50fa98765632ff3c13d64f12dbd077547785efc5015987ea
Sep 18 18:44:39.570894 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.570862    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577500df_cb0a_418e_a573_688a25c63dbb.slice/crio-3c6635ef692ba44ee7b325eda07d74b50b2bcf7ff39f808728c63d0cc829b350 WatchSource:0}: Error finding container 3c6635ef692ba44ee7b325eda07d74b50b2bcf7ff39f808728c63d0cc829b350: Status 404 returned error can't find the container with id 3c6635ef692ba44ee7b325eda07d74b50b2bcf7ff39f808728c63d0cc829b350
Sep 18 18:44:39.577746 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.577730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg"
Sep 18 18:44:39.582562 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.582537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klds4"
Sep 18 18:44:39.585301 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.585283    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6697d2f_f8b5_400f_8b9b_a59f037557a2.slice/crio-3fedaecdf3ca608b00e74c5f3d9241952909d5bef309b1c0d0d039f980020c43 WatchSource:0}: Error finding container 3fedaecdf3ca608b00e74c5f3d9241952909d5bef309b1c0d0d039f980020c43: Status 404 returned error can't find the container with id 3fedaecdf3ca608b00e74c5f3d9241952909d5bef309b1c0d0d039f980020c43
Sep 18 18:44:39.590380 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.590344    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d9edbb_99b9_44d1_81ef_9e609393c40b.slice/crio-98c5041922e930947edac738d9e5deea5f2e511a3c38655788ad3e024fcdef91 WatchSource:0}: Error finding container 98c5041922e930947edac738d9e5deea5f2e511a3c38655788ad3e024fcdef91: Status 404 returned error can't find the container with id 98c5041922e930947edac738d9e5deea5f2e511a3c38655788ad3e024fcdef91
Sep 18 18:44:39.592924 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.592870    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:44:39.616949 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.616926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hxcfx"
Sep 18 18:44:39.621565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.621549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:44:39.625223 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.625205    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37cf107_9c81_4955_8fd0_a54129f38f93.slice/crio-8d54895797d3bfbeb014b6d00fcac2cd49dc5a46a4306f139b67e44c36dea2e7 WatchSource:0}: Error finding container 8d54895797d3bfbeb014b6d00fcac2cd49dc5a46a4306f139b67e44c36dea2e7: Status 404 returned error can't find the container with id 8d54895797d3bfbeb014b6d00fcac2cd49dc5a46a4306f139b67e44c36dea2e7
Sep 18 18:44:39.629796 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.629778    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b431ce_f544_49f6_bc43_38ae2c257ae6.slice/crio-a4c016040b79a44a4ddc8dcba0357d659ad93e09edcc4b9db852a53dc377ed15 WatchSource:0}: Error finding container a4c016040b79a44a4ddc8dcba0357d659ad93e09edcc4b9db852a53dc377ed15: Status 404 returned error can't find the container with id a4c016040b79a44a4ddc8dcba0357d659ad93e09edcc4b9db852a53dc377ed15
Sep 18 18:44:39.632937 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.632922    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:44:39.637242 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.637228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hf5v7"
Sep 18 18:44:39.646017 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:44:39.645999    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba42289_c2a5_4752_87a8_f616b4372de0.slice/crio-2318d2eb54cc3f02e366035b41ffbf634701508a3e94a436ad7a9f6e3ec013d6 WatchSource:0}: Error finding container 2318d2eb54cc3f02e366035b41ffbf634701508a3e94a436ad7a9f6e3ec013d6: Status 404 returned error can't find the container with id 2318d2eb54cc3f02e366035b41ffbf634701508a3e94a436ad7a9f6e3ec013d6
Sep 18 18:44:39.829147 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.829080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:39.829147 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.829140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:39.829307 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:39.829167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:39.829307 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829271    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:39.829390 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829322    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret podName:4d0f8445-381e-4d78-b730-ac9eb826b144 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:40.829306029 +0000 UTC m=+4.621982949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret") pod "global-pull-secret-syncer-hrxqr" (UID: "4d0f8445-381e-4d78-b730-ac9eb826b144") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:39.829698 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829677    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:39.829788 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829742    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs podName:129a6d33-4815-4918-9f85-bbe434e773ea nodeName:}" failed. No retries permitted until 2026-09-18 18:44:40.829715016 +0000 UTC m=+4.622391924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs") pod "network-metrics-daemon-6xbrr" (UID: "129a6d33-4815-4918-9f85-bbe434e773ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:39.829855 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829818    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:44:39.829855 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829836    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:44:39.829855 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829849    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mlvd5 for pod openshift-network-diagnostics/network-check-target-ghpld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:39.829991 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:39.829904    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5 podName:3fe33d9a-e11a-4bb4-8535-777727b79a84 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:40.829874417 +0000 UTC m=+4.622551326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlvd5" (UniqueName: "kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5") pod "network-check-target-ghpld" (UID: "3fe33d9a-e11a-4bb4-8535-777727b79a84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:40.175693 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.175663    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:44:40.263814 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.263777    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 18:39:39 +0000 UTC" deadline="2028-03-15 19:06:07.889513685 +0000 UTC"
Sep 18 18:44:40.263814 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.263815    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13056h21m27.625703572s"
Sep 18 18:44:40.342473 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.342291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerStarted","Data":"4a3ddef44b69a85b50fa98765632ff3c13d64f12dbd077547785efc5015987ea"}
Sep 18 18:44:40.367498 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.367450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" event={"ID":"aba6b42d05a06c053c050c26af669cbb","Type":"ContainerStarted","Data":"8cb1aeb887d2d737932bb8adbb7eef524a63c101d45319abd365161143fdf0dc"}
Sep 18 18:44:40.373675 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.373645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"911c34c29a66f4fca404d6224fced36c","Type":"ContainerStarted","Data":"445f2d41df5847bb2178ae3158b092eb46b806050eeeab3afbba1481a6fd5cc9"}
Sep 18 18:44:40.389606 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.389574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hxcfx" event={"ID":"e37cf107-9c81-4955-8fd0-a54129f38f93","Type":"ContainerStarted","Data":"8d54895797d3bfbeb014b6d00fcac2cd49dc5a46a4306f139b67e44c36dea2e7"}
Sep 18 18:44:40.410926 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.410896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg" event={"ID":"d6697d2f-f8b5-400f-8b9b-a59f037557a2","Type":"ContainerStarted","Data":"3fedaecdf3ca608b00e74c5f3d9241952909d5bef309b1c0d0d039f980020c43"}
Sep 18 18:44:40.420998 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.420971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7xv6g" event={"ID":"577500df-cb0a-418e-a573-688a25c63dbb","Type":"ContainerStarted","Data":"3c6635ef692ba44ee7b325eda07d74b50b2bcf7ff39f808728c63d0cc829b350"}
Sep 18 18:44:40.429374 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.429318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-85kwk" event={"ID":"c767a1b9-30ea-4bff-801a-59c85191b563","Type":"ContainerStarted","Data":"3ac0b0e8c1d2202b0d81d831f8d386f0ebfec95197d6073ec8719bf91d40dabb"}
Sep 18 18:44:40.441715 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.441688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-78vcs" event={"ID":"9b2eac25-004e-49a9-ba24-15fe12ebfe13","Type":"ContainerStarted","Data":"8ecee8d6d74302f3c6f4ffb352d118656612b2d06852abeda9bcd5e9f0bee41a"}
Sep 18 18:44:40.453770 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.453742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hf5v7" event={"ID":"aba42289-c2a5-4752-87a8-f616b4372de0","Type":"ContainerStarted","Data":"2318d2eb54cc3f02e366035b41ffbf634701508a3e94a436ad7a9f6e3ec013d6"}
Sep 18 18:44:40.471679 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.471650    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"a4c016040b79a44a4ddc8dcba0357d659ad93e09edcc4b9db852a53dc377ed15"}
Sep 18 18:44:40.477308 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.477286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klds4" event={"ID":"d3d9edbb-99b9-44d1-81ef-9e609393c40b","Type":"ContainerStarted","Data":"98c5041922e930947edac738d9e5deea5f2e511a3c38655788ad3e024fcdef91"}
Sep 18 18:44:40.836578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.836503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:40.836578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.836556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:40.836578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:40.836577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:40.836819 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.836697    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:40.836819 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.836743    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret podName:4d0f8445-381e-4d78-b730-ac9eb826b144 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:42.836730293 +0000 UTC m=+6.629407201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret") pod "global-pull-secret-syncer-hrxqr" (UID: "4d0f8445-381e-4d78-b730-ac9eb826b144") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:40.837455 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.837166    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:40.837455 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.837247    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs podName:129a6d33-4815-4918-9f85-bbe434e773ea nodeName:}" failed. No retries permitted until 2026-09-18 18:44:42.837231924 +0000 UTC m=+6.629908848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs") pod "network-metrics-daemon-6xbrr" (UID: "129a6d33-4815-4918-9f85-bbe434e773ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:40.837455 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.837328    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:44:40.837455 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.837341    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:44:40.837455 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.837354    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mlvd5 for pod openshift-network-diagnostics/network-check-target-ghpld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:40.837455 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:40.837390    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5 podName:3fe33d9a-e11a-4bb4-8535-777727b79a84 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:42.837378777 +0000 UTC m=+6.630055701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlvd5" (UniqueName: "kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5") pod "network-check-target-ghpld" (UID: "3fe33d9a-e11a-4bb4-8535-777727b79a84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:41.264767 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:41.264685    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 18:39:39 +0000 UTC" deadline="2028-06-13 14:21:57.012771962 +0000 UTC"
Sep 18 18:44:41.264767 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:41.264738    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15211h37m15.748037403s"
Sep 18 18:44:41.329419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:41.329390    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:41.329572 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:41.329513    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:41.329960 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:41.329941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:41.330067 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:41.330047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:41.330123 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:41.330111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:41.330203 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:41.330173    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:42.283769 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:42.283738    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 18:44:42.854427 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:42.854393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:42.854600 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:42.854468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:42.854600 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:42.854499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:42.854709 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.854698    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:42.854765 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.854753    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs podName:129a6d33-4815-4918-9f85-bbe434e773ea nodeName:}" failed. No retries permitted until 2026-09-18 18:44:46.854741272 +0000 UTC m=+10.647418180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs") pod "network-metrics-daemon-6xbrr" (UID: "129a6d33-4815-4918-9f85-bbe434e773ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:42.855082 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.855066    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:44:42.855082 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.855084    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:44:42.855195 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.855093    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mlvd5 for pod openshift-network-diagnostics/network-check-target-ghpld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:42.855195 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.855124    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5 podName:3fe33d9a-e11a-4bb4-8535-777727b79a84 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:46.855114563 +0000 UTC m=+10.647791471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlvd5" (UniqueName: "kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5") pod "network-check-target-ghpld" (UID: "3fe33d9a-e11a-4bb4-8535-777727b79a84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:42.855301 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.855226    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:42.855301 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:42.855263    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret podName:4d0f8445-381e-4d78-b730-ac9eb826b144 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:46.855251527 +0000 UTC m=+10.647928449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret") pod "global-pull-secret-syncer-hrxqr" (UID: "4d0f8445-381e-4d78-b730-ac9eb826b144") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:43.329707 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:43.329675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:43.330132 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:43.329799    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:43.330132 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:43.329978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:43.330132 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:43.330099    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:43.330295 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:43.330149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:43.330295 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:43.330218    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:45.329430 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:45.329397    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:45.329980 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:45.329513    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:45.329980 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:45.329813    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:45.329980 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:45.329934    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:45.330170 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:45.329989    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:45.330170 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:45.330048    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:46.885682 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:46.885647    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:46.885716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:46.885742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885812    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885867    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885901    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885902    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885915    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mlvd5 for pod openshift-network-diagnostics/network-check-target-ghpld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885902    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs podName:129a6d33-4815-4918-9f85-bbe434e773ea nodeName:}" failed. No retries permitted until 2026-09-18 18:44:54.885865819 +0000 UTC m=+18.678542741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs") pod "network-metrics-daemon-6xbrr" (UID: "129a6d33-4815-4918-9f85-bbe434e773ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885968    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5 podName:3fe33d9a-e11a-4bb4-8535-777727b79a84 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:54.885951007 +0000 UTC m=+18.678627916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlvd5" (UniqueName: "kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5") pod "network-check-target-ghpld" (UID: "3fe33d9a-e11a-4bb4-8535-777727b79a84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:46.886188 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:46.885992    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret podName:4d0f8445-381e-4d78-b730-ac9eb826b144 nodeName:}" failed. No retries permitted until 2026-09-18 18:44:54.885983563 +0000 UTC m=+18.678660475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret") pod "global-pull-secret-syncer-hrxqr" (UID: "4d0f8445-381e-4d78-b730-ac9eb826b144") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:47.328857 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:47.328805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:47.329033 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:47.328949    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:47.329033 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:47.328971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:47.329161 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:47.329080    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:47.329161 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:47.328961    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:47.329279 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:47.329175    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:49.329601 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:49.329567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:49.330063 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:49.329583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:49.330063 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:49.329682    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:49.330063 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:49.329695    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:49.330063 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:49.329808    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:49.330063 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:49.329927    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:51.329689 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:51.329602    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:51.330130 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:51.329719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:51.330130 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:51.329746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:51.330130 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:51.329734    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:51.330130 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:51.329834    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:51.330130 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:51.329923    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:53.329098 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:53.329063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:53.329521 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:53.329061    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:53.329521 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:53.329193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:53.329521 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:53.329256    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:53.329521 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:53.329063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:53.329825 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:53.329723    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:54.945444 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:54.945410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:54.945476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:54.945500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945576    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945587    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945604    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945620    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945630    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mlvd5 for pod openshift-network-diagnostics/network-check-target-ghpld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945645    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs podName:129a6d33-4815-4918-9f85-bbe434e773ea nodeName:}" failed. No retries permitted until 2026-09-18 18:45:10.94562548 +0000 UTC m=+34.738302406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs") pod "network-metrics-daemon-6xbrr" (UID: "129a6d33-4815-4918-9f85-bbe434e773ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945666    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5 podName:3fe33d9a-e11a-4bb4-8535-777727b79a84 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:10.945654988 +0000 UTC m=+34.738331899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlvd5" (UniqueName: "kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5") pod "network-check-target-ghpld" (UID: "3fe33d9a-e11a-4bb4-8535-777727b79a84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:44:54.945875 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:54.945679    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret podName:4d0f8445-381e-4d78-b730-ac9eb826b144 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:10.945673557 +0000 UTC m=+34.738350465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret") pod "global-pull-secret-syncer-hrxqr" (UID: "4d0f8445-381e-4d78-b730-ac9eb826b144") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:44:55.329133 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:55.329052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:55.329133 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:55.329099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:55.329133 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:55.329099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:55.329433 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:55.329176    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:55.329433 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:55.329273    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:55.329433 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:55.329328    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:57.329502 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:57.329469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:57.329942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:57.329586    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:57.329942 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:57.329596    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:57.329942 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:57.329691    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:57.329942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:57.329732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:57.329942 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:57.329802    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:59.329515 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.329320    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:44:59.330028 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.329320    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:44:59.330028 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:59.329543    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:44:59.330028 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:59.329630    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:44:59.330028 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.329345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:44:59.330028 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:44:59.329700    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:44:59.550320 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.550291    2582 generic.go:356] "Generic (PLEG): container finished" podID="911c34c29a66f4fca404d6224fced36c" containerID="c28571b4b515f8058df49154972392df380f548f37660ccfb934109c8fc20e4e" exitCode=0
Sep 18 18:44:59.550447 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.550386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"911c34c29a66f4fca404d6224fced36c","Type":"ContainerDied","Data":"c28571b4b515f8058df49154972392df380f548f37660ccfb934109c8fc20e4e"}
Sep 18 18:44:59.552727 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.552703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg" event={"ID":"d6697d2f-f8b5-400f-8b9b-a59f037557a2","Type":"ContainerStarted","Data":"8b213c599d5c5285fe8f94958b60331a718f99018369291134cac226dcfbff6e"}
Sep 18 18:44:59.554374 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.554348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7xv6g" event={"ID":"577500df-cb0a-418e-a573-688a25c63dbb","Type":"ContainerStarted","Data":"1d9770014ba628ac72f09ddcb75568da594fdb6fe6f2a068f9804620e7e2eee8"}
Sep 18 18:44:59.556289 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.556265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-85kwk" event={"ID":"c767a1b9-30ea-4bff-801a-59c85191b563","Type":"ContainerStarted","Data":"aee29c05ea55525af2e15af3600845cbc156f6ef63819d5ca5fc79f893ed5fab"}
Sep 18 18:44:59.557855 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.557830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-78vcs" event={"ID":"9b2eac25-004e-49a9-ba24-15fe12ebfe13","Type":"ContainerStarted","Data":"32648da88065caf63718cecffbaa8b6c691b00befe8293a839ee0eacfa13ee53"}
Sep 18 18:44:59.559496 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.559472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hf5v7" event={"ID":"aba42289-c2a5-4752-87a8-f616b4372de0","Type":"ContainerStarted","Data":"2180343a929b9c0dd91fb655e06a0bad1f020226b6854bf753c34d68dbbee4dc"}
Sep 18 18:44:59.562910 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.562870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"166da2872c4621efd9e774a13ca7a02ec363f8ef6a54b4fc1b70dce11a62ae1b"}
Sep 18 18:44:59.562999 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.562962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"1aa45a7ab79044fff68762a8b0d56979caac50ed1ace3feff812f97f43de009d"}
Sep 18 18:44:59.562999 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.562996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"662914da34cc3c107f347c1795e9932def8a093611e320dc259d1af3d90400da"}
Sep 18 18:44:59.563107 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.563033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"47fa796f08cce83a8889632721572b0ab9bb58cd99c49cf3d4e8888e9a2eb99c"}
Sep 18 18:44:59.563107 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.563063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"675652bbcf82af7efdd9f284bf07bf37f4735de814c734a6380ff908313888ea"}
Sep 18 18:44:59.563107 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.563097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"7a1d6ae761d203d9aebcc2bebb22e6f69953ec23a61e134729eaac4dfd0c14e8"}
Sep 18 18:44:59.564335 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.564317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klds4" event={"ID":"d3d9edbb-99b9-44d1-81ef-9e609393c40b","Type":"ContainerStarted","Data":"7885ccd83d937a1969da59ced69eb261a26ec57fbb2f5d9969f3c238e0bf79f0"}
Sep 18 18:44:59.566146 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.566119    2582 generic.go:356] "Generic (PLEG): container finished" podID="a87e2cb4-40dc-4b7a-ba56-2d48085367f1" containerID="7c4a76a097b91ae947afe774b7fd0e31a94314158c8cf9bf9066e49fe138520a" exitCode=0
Sep 18 18:44:59.566242 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.566182    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerDied","Data":"7c4a76a097b91ae947afe774b7fd0e31a94314158c8cf9bf9066e49fe138520a"}
Sep 18 18:44:59.568007 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.567989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" event={"ID":"aba6b42d05a06c053c050c26af669cbb","Type":"ContainerStarted","Data":"f3c5598f4746324443d09e6dde527e61b795edf0e724056b0a03aad04645e5c4"}
Sep 18 18:44:59.571012 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.570968    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-168.ec2.internal" podStartSLOduration=20.570956308 podStartE2EDuration="20.570956308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:58.559463006 +0000 UTC m=+22.352139936" watchObservedRunningTime="2026-09-18 18:44:59.570956308 +0000 UTC m=+23.363633237"
Sep 18 18:44:59.590564 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.590520    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hf5v7" podStartSLOduration=2.8232502520000002 podStartE2EDuration="21.590505172s" totalImagesPullingTime="18.76725492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:59.590088221 +0000 UTC m=+23.382765175" watchObservedRunningTime="2026-09-18 18:44:59.590505172 +0000 UTC m=+23.383182093"
Sep 18 18:44:59.617027 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.616985    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-85kwk" podStartSLOduration=2.698421252 podStartE2EDuration="21.616969714s" totalImagesPullingTime="18.918548462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:59.616806682 +0000 UTC m=+23.409483631" watchObservedRunningTime="2026-09-18 18:44:59.616969714 +0000 UTC m=+23.409646645"
Sep 18 18:44:59.657897 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.657854    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7xv6g" podStartSLOduration=2.838073562 podStartE2EDuration="21.657846311s" totalImagesPullingTime="18.819772749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:59.657182087 +0000 UTC m=+23.449859016" watchObservedRunningTime="2026-09-18 18:44:59.657846311 +0000 UTC m=+23.450523241"
Sep 18 18:44:59.678645 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.678609    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-78vcs" podStartSLOduration=2.79706432 podStartE2EDuration="21.678592596s" totalImagesPullingTime="18.881528276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:59.678297096 +0000 UTC m=+23.470974046" watchObservedRunningTime="2026-09-18 18:44:59.678592596 +0000 UTC m=+23.471269526"
Sep 18 18:44:59.710289 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.710120    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-klds4" podStartSLOduration=2.885884607 podStartE2EDuration="21.710108211s" totalImagesPullingTime="18.824223604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:44:59.709779488 +0000 UTC m=+23.502456412" watchObservedRunningTime="2026-09-18 18:44:59.710108211 +0000 UTC m=+23.502785139"
Sep 18 18:44:59.798021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.797986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:44:59.878962 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:44:59.878841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:00.179798 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.179630    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 18:45:00.299679 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.299589    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T18:45:00.179643699Z","UUID":"0018f6b4-8c9a-4208-8924-10aa4a927286","Handler":null,"Name":"","Endpoint":""}
Sep 18 18:45:00.301045 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.301026    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 18:45:00.301045 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.301058    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 18:45:00.571902 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.571821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" event={"ID":"911c34c29a66f4fca404d6224fced36c","Type":"ContainerStarted","Data":"5fd139cc9ab4ffca42078cd8f6551f583e866ad0c70f7593e9afc42af77914ef"}
Sep 18 18:45:00.573305 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.573284    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hxcfx" event={"ID":"e37cf107-9c81-4955-8fd0-a54129f38f93","Type":"ContainerStarted","Data":"e226d98933e04b7a95863687315ff1f3acbde68ae9d1e302db63284bc87cc777"}
Sep 18 18:45:00.575088 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.575063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg" event={"ID":"d6697d2f-f8b5-400f-8b9b-a59f037557a2","Type":"ContainerStarted","Data":"fbcd7aa7edcab125015bca12f9b73e42c16fb71502a861fcc2850b290e9c2788"}
Sep 18 18:45:00.588142 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.588101    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-168.ec2.internal" podStartSLOduration=21.588087392 podStartE2EDuration="21.588087392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:00.587612266 +0000 UTC m=+24.380289221" watchObservedRunningTime="2026-09-18 18:45:00.588087392 +0000 UTC m=+24.380764313"
Sep 18 18:45:00.606307 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.606266    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hxcfx" podStartSLOduration=3.840967264 podStartE2EDuration="22.606254165s" totalImagesPullingTime="18.765286901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:00.605648658 +0000 UTC m=+24.398325619" watchObservedRunningTime="2026-09-18 18:45:00.606254165 +0000 UTC m=+24.398931098"
Sep 18 18:45:00.725369 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.725342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:00.804242 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:00.804214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:01.215986 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.215961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:01.318277 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.318217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:01.329285 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.329262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:01.329392 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.329285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:01.329392 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.329298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:01.329392 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:01.329362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:45:01.329534 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:01.329458    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:45:01.329585 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:01.329535    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:45:01.579160 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.579125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg" event={"ID":"d6697d2f-f8b5-400f-8b9b-a59f037557a2","Type":"ContainerStarted","Data":"7e84140c0085d664af4d0a72fb4c03e533bcdefdf9750fbabbcafb80b5ba1d46"}
Sep 18 18:45:01.582525 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.582499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"45e4fcca907bc3978de212c12d4004487d3b2e97df2fdcf8d44ac61aadfc0c5f"}
Sep 18 18:45:01.608591 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:01.608553    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xktkg" podStartSLOduration=2.425720156 podStartE2EDuration="23.608540694s" totalImagesPullingTime="21.182820538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:01.608454416 +0000 UTC m=+25.401131360" watchObservedRunningTime="2026-09-18 18:45:01.608540694 +0000 UTC m=+25.401217649"
Sep 18 18:45:02.216025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:02.215989    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:45:02.216574 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:02.216554    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:45:02.584486 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:02.584356    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:45:02.584837 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:02.584653    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7xv6g"
Sep 18 18:45:02.681064 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:02.681037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:02.744001 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:02.743973    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:03.329184 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:03.329155    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:03.329347 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:03.329190    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:03.329347 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:03.329258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:03.329347 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:03.329271    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:45:03.329347 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:03.329338    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:45:03.329518 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:03.329405    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:45:03.398403 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:03.398373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:03.470399 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:03.470371    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:04.592286 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:04.592083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" event={"ID":"99b431ce-f544-49f6-bc43-38ae2c257ae6","Type":"ContainerStarted","Data":"1745ef6e29132661a83e7b8798d37bb779fcff0b886c3368a5c6a09e1b39539c"}
Sep 18 18:45:04.616504 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:04.616453    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl" podStartSLOduration=7.770453374 podStartE2EDuration="26.61643898s" totalImagesPullingTime="18.845985606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:04.61554454 +0000 UTC m=+28.408221503" watchObservedRunningTime="2026-09-18 18:45:04.61643898 +0000 UTC m=+28.409115909"
Sep 18 18:45:05.329131 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.329104    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:05.329131 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.329124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:05.329250 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:05.329193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:45:05.329250 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.329239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:05.329343 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:05.329325    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:45:05.329409 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:05.329393    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:45:05.544338 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.544306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:05.599344 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.598193    2582 generic.go:356] "Generic (PLEG): container finished" podID="a87e2cb4-40dc-4b7a-ba56-2d48085367f1" containerID="07720110b560b81fe1062833e0ec21d0bbabd66290397ff847120856bcb57b69" exitCode=0
Sep 18 18:45:05.599344 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.598749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerDied","Data":"07720110b560b81fe1062833e0ec21d0bbabd66290397ff847120856bcb57b69"}
Sep 18 18:45:05.599946 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.599527    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:45:05.599946 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.599592    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:45:05.600718 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.600701    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:45:05.631474 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.631449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:45:05.632106 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.632084    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:45:05.650389 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:05.650352    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:06.441657 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.441627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:06.484590 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.484568    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxqr"]
Sep 18 18:45:06.484679 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.484661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:06.484762 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:06.484746    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:45:06.496107 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.495895    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xbrr"]
Sep 18 18:45:06.496107 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.495983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:06.496107 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:06.496077    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:45:06.506501 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.506475    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ghpld"]
Sep 18 18:45:06.506611 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.506544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:06.506667 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:06.506609    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:45:06.565647 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.565487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:06.602336 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:06.602313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerStarted","Data":"d535720d7b91135147b121e17a522ae4d939e4bd735a5fc1667edd40987d9d6b"}
Sep 18 18:45:07.606440 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:07.606413    2582 generic.go:356] "Generic (PLEG): container finished" podID="a87e2cb4-40dc-4b7a-ba56-2d48085367f1" containerID="d535720d7b91135147b121e17a522ae4d939e4bd735a5fc1667edd40987d9d6b" exitCode=0
Sep 18 18:45:07.606776 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:07.606496    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerDied","Data":"d535720d7b91135147b121e17a522ae4d939e4bd735a5fc1667edd40987d9d6b"}
Sep 18 18:45:08.329982 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:08.329960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:08.330058 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:08.330037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:08.330102 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:08.330067    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:45:08.330137 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:08.330114    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:45:08.330169 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:08.330129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:08.330236 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:08.330221    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:45:08.612385 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:08.612364    2582 generic.go:356] "Generic (PLEG): container finished" podID="a87e2cb4-40dc-4b7a-ba56-2d48085367f1" containerID="93b67f35257c640aaadd845cf2a6889fdeee755361773c5289c07643fadfaf41" exitCode=0
Sep 18 18:45:08.612659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:08.612390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerDied","Data":"93b67f35257c640aaadd845cf2a6889fdeee755361773c5289c07643fadfaf41"}
Sep 18 18:45:09.755374 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:09.755335    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:09.851013 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:09.850981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:10.332356 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.332284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:10.332494 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.332390    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:10.332494 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.332407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:10.332494 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.332401    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqr" podUID="4d0f8445-381e-4d78-b730-ac9eb826b144"
Sep 18 18:45:10.332494 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.332473    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ghpld" podUID="3fe33d9a-e11a-4bb4-8535-777727b79a84"
Sep 18 18:45:10.332655 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.332568    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xbrr" podUID="129a6d33-4815-4918-9f85-bbe434e773ea"
Sep 18 18:45:10.345022 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.345000    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:10.435554 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.435531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:10.969005 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.968971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.969042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:10.969073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969137    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969184    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969212    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs podName:129a6d33-4815-4918-9f85-bbe434e773ea nodeName:}" failed. No retries permitted until 2026-09-18 18:45:42.969191633 +0000 UTC m=+66.761868565 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs") pod "network-metrics-daemon-6xbrr" (UID: "129a6d33-4815-4918-9f85-bbe434e773ea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969237    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret podName:4d0f8445-381e-4d78-b730-ac9eb826b144 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:42.969220011 +0000 UTC m=+66.761896925 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret") pod "global-pull-secret-syncer-hrxqr" (UID: "4d0f8445-381e-4d78-b730-ac9eb826b144") : object "kube-system"/"original-pull-secret" not registered
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969261    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969278    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969289    2582 projected.go:196] Error preparing data for projected volume kube-api-access-mlvd5 for pod openshift-network-diagnostics/network-check-target-ghpld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:45:10.969419 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:10.969363    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5 podName:3fe33d9a-e11a-4bb4-8535-777727b79a84 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:42.969348662 +0000 UTC m=+66.762025590 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlvd5" (UniqueName: "kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5") pod "network-check-target-ghpld" (UID: "3fe33d9a-e11a-4bb4-8535-777727b79a84") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 18:45:12.193077 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.192999    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-168.ec2.internal" event="NodeReady"
Sep 18 18:45:12.193559 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.193114    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 18:45:12.243988 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.243953    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j54tt"]
Sep 18 18:45:12.277282 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.277246    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kbzzh"]
Sep 18 18:45:12.277417 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.277366    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.280290 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.280258    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 18:45:12.280511 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.280494    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 18:45:12.280603 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.280499    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qf4nw\""
Sep 18 18:45:12.291209 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.291189    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kbzzh"]
Sep 18 18:45:12.291292 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.291216    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j54tt"]
Sep 18 18:45:12.291292 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.291282    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.294598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.294368    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 18:45:12.294598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.294413    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 18:45:12.294598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.294424    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 18:45:12.294598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.294458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sl7vh\""
Sep 18 18:45:12.329295 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.329274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:12.329415 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.329294    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:12.329415 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.329283    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:12.335268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.333470    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 18:45:12.335268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.333456    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 18:45:12.335268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.334356    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 18:45:12.335268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.334285    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vk59r\""
Sep 18 18:45:12.335268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.335064    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 18:45:12.335857 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.335834    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdsv4\""
Sep 18 18:45:12.377316 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.377293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh54m\" (UniqueName: \"kubernetes.io/projected/2c7ee963-be99-40a0-821d-4d8fc461a0c6-kube-api-access-mh54m\") pod \"ingress-canary-kbzzh\" (UID: \"2c7ee963-be99-40a0-821d-4d8fc461a0c6\") " pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.377410 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.377325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ee963-be99-40a0-821d-4d8fc461a0c6-cert\") pod \"ingress-canary-kbzzh\" (UID: \"2c7ee963-be99-40a0-821d-4d8fc461a0c6\") " pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.377410 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.377380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.377517 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.377446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b41324a-3369-42ea-aa62-aa5b19ce808e-config-volume\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.377517 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.377492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b41324a-3369-42ea-aa62-aa5b19ce808e-tmp-dir\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.377623 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.377546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nndw\" (UniqueName: \"kubernetes.io/projected/8b41324a-3369-42ea-aa62-aa5b19ce808e-kube-api-access-7nndw\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.392800 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.392769    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:12.460554 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.460499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:12.478496 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.478477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.478580 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.478504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b41324a-3369-42ea-aa62-aa5b19ce808e-config-volume\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.478580 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.478529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b41324a-3369-42ea-aa62-aa5b19ce808e-tmp-dir\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.478690 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.478583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nndw\" (UniqueName: \"kubernetes.io/projected/8b41324a-3369-42ea-aa62-aa5b19ce808e-kube-api-access-7nndw\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.478690 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:12.478589    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:45:12.478690 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.478628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh54m\" (UniqueName: \"kubernetes.io/projected/2c7ee963-be99-40a0-821d-4d8fc461a0c6-kube-api-access-mh54m\") pod \"ingress-canary-kbzzh\" (UID: \"2c7ee963-be99-40a0-821d-4d8fc461a0c6\") " pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.478690 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:12.478655    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls podName:8b41324a-3369-42ea-aa62-aa5b19ce808e nodeName:}" failed. No retries permitted until 2026-09-18 18:45:12.978633664 +0000 UTC m=+36.771310573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls") pod "dns-default-j54tt" (UID: "8b41324a-3369-42ea-aa62-aa5b19ce808e") : secret "dns-default-metrics-tls" not found
Sep 18 18:45:12.478904 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.478693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ee963-be99-40a0-821d-4d8fc461a0c6-cert\") pod \"ingress-canary-kbzzh\" (UID: \"2c7ee963-be99-40a0-821d-4d8fc461a0c6\") " pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.478904 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.478854    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b41324a-3369-42ea-aa62-aa5b19ce808e-tmp-dir\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.479153 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.479130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b41324a-3369-42ea-aa62-aa5b19ce808e-config-volume\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.483664 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.483639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ee963-be99-40a0-821d-4d8fc461a0c6-cert\") pod \"ingress-canary-kbzzh\" (UID: \"2c7ee963-be99-40a0-821d-4d8fc461a0c6\") " pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.487556 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.487519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh54m\" (UniqueName: \"kubernetes.io/projected/2c7ee963-be99-40a0-821d-4d8fc461a0c6-kube-api-access-mh54m\") pod \"ingress-canary-kbzzh\" (UID: \"2c7ee963-be99-40a0-821d-4d8fc461a0c6\") " pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.487786 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.487763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nndw\" (UniqueName: \"kubernetes.io/projected/8b41324a-3369-42ea-aa62-aa5b19ce808e-kube-api-access-7nndw\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.601341 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.601311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kbzzh"
Sep 18 18:45:12.983322 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:12.983168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:12.983442 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:12.983335    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:45:12.983442 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:12.983398    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls podName:8b41324a-3369-42ea-aa62-aa5b19ce808e nodeName:}" failed. No retries permitted until 2026-09-18 18:45:13.98338322 +0000 UTC m=+37.776060141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls") pod "dns-default-j54tt" (UID: "8b41324a-3369-42ea-aa62-aa5b19ce808e") : secret "dns-default-metrics-tls" not found
Sep 18 18:45:13.078926 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:13.078904    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kbzzh"]
Sep 18 18:45:13.082161 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:13.082121    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7ee963_be99_40a0_821d_4d8fc461a0c6.slice/crio-6a0d58a9c94fcdc3e23f0a0c01c5ec6241920682ec3191c544cac676dff66d26 WatchSource:0}: Error finding container 6a0d58a9c94fcdc3e23f0a0c01c5ec6241920682ec3191c544cac676dff66d26: Status 404 returned error can't find the container with id 6a0d58a9c94fcdc3e23f0a0c01c5ec6241920682ec3191c544cac676dff66d26
Sep 18 18:45:13.624734 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:13.624700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kbzzh" event={"ID":"2c7ee963-be99-40a0-821d-4d8fc461a0c6","Type":"ContainerStarted","Data":"6a0d58a9c94fcdc3e23f0a0c01c5ec6241920682ec3191c544cac676dff66d26"}
Sep 18 18:45:13.627369 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:13.627346    2582 generic.go:356] "Generic (PLEG): container finished" podID="a87e2cb4-40dc-4b7a-ba56-2d48085367f1" containerID="6e449b8fa7c21a8d2de2e2bde166ab21e4335427fa664840ca3cb4fdc5e6fc94" exitCode=0
Sep 18 18:45:13.627467 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:13.627388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerDied","Data":"6e449b8fa7c21a8d2de2e2bde166ab21e4335427fa664840ca3cb4fdc5e6fc94"}
Sep 18 18:45:13.696241 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:13.696216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:13.760807 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:13.760787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:13.991136 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:13.991107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:13.991288 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:13.991221    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:45:13.991288 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:13.991286    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls podName:8b41324a-3369-42ea-aa62-aa5b19ce808e nodeName:}" failed. No retries permitted until 2026-09-18 18:45:15.991270779 +0000 UTC m=+39.783947691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls") pod "dns-default-j54tt" (UID: "8b41324a-3369-42ea-aa62-aa5b19ce808e") : secret "dns-default-metrics-tls" not found
Sep 18 18:45:14.632393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:14.632354    2582 generic.go:356] "Generic (PLEG): container finished" podID="a87e2cb4-40dc-4b7a-ba56-2d48085367f1" containerID="9fdb7c59dbaca601ab65464109ba3d3fc0639a16c780c020af6d9c2a184ace5b" exitCode=0
Sep 18 18:45:14.632393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:14.632390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerDied","Data":"9fdb7c59dbaca601ab65464109ba3d3fc0639a16c780c020af6d9c2a184ace5b"}
Sep 18 18:45:14.976603 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:14.976572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:15.083560 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.083515    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:15.637478 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.637446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vszhp" event={"ID":"a87e2cb4-40dc-4b7a-ba56-2d48085367f1","Type":"ContainerStarted","Data":"f274a83c71f8b26d0d02c15b9a7e813fc268e7b55d2e5507b2ea758578305b8d"}
Sep 18 18:45:15.660066 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.660019    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vszhp" podStartSLOduration=7.932071562 podStartE2EDuration="37.660003557s" totalImagesPullingTime="29.727931995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:15.659379342 +0000 UTC m=+39.452056293" watchObservedRunningTime="2026-09-18 18:45:15.660003557 +0000 UTC m=+39.452680487"
Sep 18 18:45:15.785158 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.785100    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:15.862691 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.862668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:15.962068 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.961861    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"]
Sep 18 18:45:15.994576 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.994559    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"]
Sep 18 18:45:15.994661 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.994638    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"
Sep 18 18:45:15.997829 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.997808    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cpwfp\""
Sep 18 18:45:15.998155 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.998129    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 18:45:15.998365 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:15.998348    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 18:45:16.009224 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.009206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:16.009366 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:16.009348    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:45:16.009433 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:16.009412    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls podName:8b41324a-3369-42ea-aa62-aa5b19ce808e nodeName:}" failed. No retries permitted until 2026-09-18 18:45:20.009393944 +0000 UTC m=+43.802070858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls") pod "dns-default-j54tt" (UID: "8b41324a-3369-42ea-aa62-aa5b19ce808e") : secret "dns-default-metrics-tls" not found
Sep 18 18:45:16.035771 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.035704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:16.109929 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.109876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rzx\" (UniqueName: \"kubernetes.io/projected/8e6b93e6-7844-4d05-81fa-1494074609d8-kube-api-access-47rzx\") pod \"migrator-76794cbd6c-nbjxb\" (UID: \"8e6b93e6-7844-4d05-81fa-1494074609d8\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"
Sep 18 18:45:16.133126 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.133099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:16.210542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.210519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rzx\" (UniqueName: \"kubernetes.io/projected/8e6b93e6-7844-4d05-81fa-1494074609d8-kube-api-access-47rzx\") pod \"migrator-76794cbd6c-nbjxb\" (UID: \"8e6b93e6-7844-4d05-81fa-1494074609d8\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"
Sep 18 18:45:16.223082 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.223061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rzx\" (UniqueName: \"kubernetes.io/projected/8e6b93e6-7844-4d05-81fa-1494074609d8-kube-api-access-47rzx\") pod \"migrator-76794cbd6c-nbjxb\" (UID: \"8e6b93e6-7844-4d05-81fa-1494074609d8\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"
Sep 18 18:45:16.302454 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.302402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"
Sep 18 18:45:16.565178 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.565112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:16.606824 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.606799    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb"]
Sep 18 18:45:16.610443 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:16.610416    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6b93e6_7844_4d05_81fa_1494074609d8.slice/crio-bbcba8ee54cebc25bb14e3e89be68b8ece3ed19ea0020425028fe58c457a3558 WatchSource:0}: Error finding container bbcba8ee54cebc25bb14e3e89be68b8ece3ed19ea0020425028fe58c457a3558: Status 404 returned error can't find the container with id bbcba8ee54cebc25bb14e3e89be68b8ece3ed19ea0020425028fe58c457a3558
Sep 18 18:45:16.644327 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.644301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kbzzh" event={"ID":"2c7ee963-be99-40a0-821d-4d8fc461a0c6","Type":"ContainerStarted","Data":"ba76f7cbdd4d55644f9c550adf93dc1d2652aa5e16da263217bc42fae6126910"}
Sep 18 18:45:16.645702 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.645680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb" event={"ID":"8e6b93e6-7844-4d05-81fa-1494074609d8","Type":"ContainerStarted","Data":"bbcba8ee54cebc25bb14e3e89be68b8ece3ed19ea0020425028fe58c457a3558"}
Sep 18 18:45:16.682086 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.682047    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kbzzh" podStartSLOduration=1.883268992 podStartE2EDuration="4.682035092s" totalImagesPullingTime="2.7987661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:16.681659402 +0000 UTC m=+40.474336345" watchObservedRunningTime="2026-09-18 18:45:16.682035092 +0000 UTC m=+40.474712022"
Sep 18 18:45:16.709294 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:16.709268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:17.498164 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.498133    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:17.511056 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.511031    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2twxf"]
Sep 18 18:45:17.525827 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.525807    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.528347 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.528329    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 18:45:17.528451 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.528421    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zff62\""
Sep 18 18:45:17.528657 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.528641    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 18:45:17.529093 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.529080    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 18:45:17.529134 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.529101    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 18:45:17.531376 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.531356    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2twxf"]
Sep 18 18:45:17.590971 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.590947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:17.621640 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.621621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f46265-978a-4df5-a719-87f818fe09a1-data-volume\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.621717 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.621656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.621717 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.621674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5jb\" (UniqueName: \"kubernetes.io/projected/03f46265-978a-4df5-a719-87f818fe09a1-kube-api-access-8j5jb\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.621717 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.621692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03f46265-978a-4df5-a719-87f818fe09a1-crio-socket\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.621717 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.621711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.722485 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.722461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f46265-978a-4df5-a719-87f818fe09a1-data-volume\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.722507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.722528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5jb\" (UniqueName: \"kubernetes.io/projected/03f46265-978a-4df5-a719-87f818fe09a1-kube-api-access-8j5jb\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.722618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03f46265-978a-4df5-a719-87f818fe09a1-crio-socket\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:17.722629    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.722645    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:17.722695    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:18.222676594 +0000 UTC m=+42.015353522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:17.722723    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:17.722782    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:18.222765591 +0000 UTC m=+42.015442516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : configmap "kube-rbac-proxy" not found
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.722789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f46265-978a-4df5-a719-87f818fe09a1-data-volume\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.722916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.722877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03f46265-978a-4df5-a719-87f818fe09a1-crio-socket\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.734318 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.734298    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5jb\" (UniqueName: \"kubernetes.io/projected/03f46265-978a-4df5-a719-87f818fe09a1-kube-api-access-8j5jb\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:17.838444 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.838386    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:17.939138 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:17.939114    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:18.225428 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.225406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:18.225504 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.225435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:18.225550 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:18.225512    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:45:18.225550 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:18.225536    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:18.225610 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:18.225556    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:19.225544992 +0000 UTC m=+43.018221913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : configmap "kube-rbac-proxy" not found
Sep 18 18:45:18.225610 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:18.225581    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:19.225568205 +0000 UTC m=+43.018245117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:18.535989 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.535933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:18.592952 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.592927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:18.651830 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.651806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb" event={"ID":"8e6b93e6-7844-4d05-81fa-1494074609d8","Type":"ContainerStarted","Data":"ed6f90cb4894cce021ddf87d5053b65d16aba2f75acd0b5ae93e911b127892df"}
Sep 18 18:45:18.651959 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.651874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb" event={"ID":"8e6b93e6-7844-4d05-81fa-1494074609d8","Type":"ContainerStarted","Data":"c0665ec42ad33e442472e81ea6fda01f95c334b95b789400fd0b990a04e08a4a"}
Sep 18 18:45:18.668261 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.668216    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-nbjxb" podStartSLOduration=2.042607792 podStartE2EDuration="3.668203004s" totalImagesPullingTime="1.625595212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:18.66798074 +0000 UTC m=+42.460657682" watchObservedRunningTime="2026-09-18 18:45:18.668203004 +0000 UTC m=+42.460879934"
Sep 18 18:45:18.745607 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.745585    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:18.811774 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:18.811726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:19.232403 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:19.232385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:19.232501 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:19.232413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:19.232548 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:19.232503    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:45:19.232548 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:19.232510    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:19.232548 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:19.232542    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:21.232526768 +0000 UTC m=+45.025203681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : configmap "kube-rbac-proxy" not found
Sep 18 18:45:19.232648 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:19.232560    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:21.232549085 +0000 UTC m=+45.025225997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:20.036425 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.036398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:20.036683 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:20.036525    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 18:45:20.036683 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:20.036578    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls podName:8b41324a-3369-42ea-aa62-aa5b19ce808e nodeName:}" failed. No retries permitted until 2026-09-18 18:45:28.036562547 +0000 UTC m=+51.829239472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls") pod "dns-default-j54tt" (UID: "8b41324a-3369-42ea-aa62-aa5b19ce808e") : secret "dns-default-metrics-tls" not found
Sep 18 18:45:20.134533 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.134510    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-29z9c"]
Sep 18 18:45:20.152324 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.152301    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.152437 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.152322    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-29z9c"]
Sep 18 18:45:20.154428 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.154407    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 18:45:20.154538 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.154432    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 18:45:20.154538 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.154439    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 18:45:20.154538 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.154443    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 18:45:20.155166 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.155154    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-txdfs\""
Sep 18 18:45:20.175694 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.175669    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:20.234293 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.234273    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hf5v7_aba42289-c2a5-4752-87a8-f616b4372de0/dns-node-resolver/0.log"
Sep 18 18:45:20.237571 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.237554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dldd\" (UniqueName: \"kubernetes.io/projected/d6c165d8-3540-451c-b578-d5a4952b510d-kube-api-access-6dldd\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.237626 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.237598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6c165d8-3540-451c-b578-d5a4952b510d-signing-cabundle\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.237685 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.237651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6c165d8-3540-451c-b578-d5a4952b510d-signing-key\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.251612 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.251591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:20.338460 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.338420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dldd\" (UniqueName: \"kubernetes.io/projected/d6c165d8-3540-451c-b578-d5a4952b510d-kube-api-access-6dldd\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.338530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.338463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6c165d8-3540-451c-b578-d5a4952b510d-signing-cabundle\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.338530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.338490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6c165d8-3540-451c-b578-d5a4952b510d-signing-key\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.339076 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.339052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6c165d8-3540-451c-b578-d5a4952b510d-signing-cabundle\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.340753 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.340735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6c165d8-3540-451c-b578-d5a4952b510d-signing-key\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.348429 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.348405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dldd\" (UniqueName: \"kubernetes.io/projected/d6c165d8-3540-451c-b578-d5a4952b510d-kube-api-access-6dldd\") pod \"service-ca-55f6fc7b86-29z9c\" (UID: \"d6c165d8-3540-451c-b578-d5a4952b510d\") " pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.460171 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.460154    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c"
Sep 18 18:45:20.764676 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:20.764647    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-29z9c"]
Sep 18 18:45:20.767927 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:20.767903    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c165d8_3540_451c_b578_d5a4952b510d.slice/crio-c5b2517efa0cdce0668ad62e5048c0c50c36969316b6a3af81ffb22429fa96af WatchSource:0}: Error finding container c5b2517efa0cdce0668ad62e5048c0c50c36969316b6a3af81ffb22429fa96af: Status 404 returned error can't find the container with id c5b2517efa0cdce0668ad62e5048c0c50c36969316b6a3af81ffb22429fa96af
Sep 18 18:45:21.032042 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:21.032000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-78vcs_9b2eac25-004e-49a9-ba24-15fe12ebfe13/node-ca/0.log"
Sep 18 18:45:21.244514 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:21.244493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:21.244774 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:21.244523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:21.244774 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:21.244615    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:45:21.244774 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:21.244641    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:21.244774 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:21.244659    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:25.244647913 +0000 UTC m=+49.037324821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : configmap "kube-rbac-proxy" not found
Sep 18 18:45:21.244774 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:21.244687    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:25.244675267 +0000 UTC m=+49.037352176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:21.277994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:21.277966    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:21.660950 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:21.660913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c" event={"ID":"d6c165d8-3540-451c-b578-d5a4952b510d","Type":"ContainerStarted","Data":"c5b2517efa0cdce0668ad62e5048c0c50c36969316b6a3af81ffb22429fa96af"}
Sep 18 18:45:21.832661 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:21.832635    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kbzzh_2c7ee963-be99-40a0-821d-4d8fc461a0c6/serve-healthcheck-canary/0.log"
Sep 18 18:45:22.432419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:22.432392    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-nbjxb_8e6b93e6-7844-4d05-81fa-1494074609d8/migrator/0.log"
Sep 18 18:45:22.632192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:22.632165    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-nbjxb_8e6b93e6-7844-4d05-81fa-1494074609d8/graceful-termination/0.log"
Sep 18 18:45:23.666391 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:23.666362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c" event={"ID":"d6c165d8-3540-451c-b578-d5a4952b510d","Type":"ContainerStarted","Data":"34f34f808dede5e5cf04c4b531942e3c162d19bc9c0252288291b55b3411c64f"}
Sep 18 18:45:23.686558 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:23.686471    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-29z9c" podStartSLOduration=1.539299303 podStartE2EDuration="3.686457183s" totalImagesPullingTime="2.14715788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:23.685155531 +0000 UTC m=+47.477832474" watchObservedRunningTime="2026-09-18 18:45:23.686457183 +0000 UTC m=+47.479134127"
Sep 18 18:45:24.621104 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:24.621061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:24.724742 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:24.724699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:24.875323 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:24.875255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:24.974897 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:24.974862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:25.079124 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:25.079103    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:25.212185 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:25.212168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:25.273941 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:25.273915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:25.274031 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:25.273961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:25.274031 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:25.274014    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 18:45:25.274031 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:25.274018    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:25.274182 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:25.274073    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:33.274057001 +0000 UTC m=+57.066733924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : configmap "kube-rbac-proxy" not found
Sep 18 18:45:25.274182 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:25.274090    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:33.274080926 +0000 UTC m=+57.066757843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:25.309613 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:25.309585    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:25.711176 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:25.711146    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:25.926064 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:25.926042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:26.516711 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:26.516680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:27.118964 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:27.118942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:27.387704 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:27.387624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:27.718150 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:27.718127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:27.905366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:27.905331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:28.095941 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.095875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:28.098052 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.098036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b41324a-3369-42ea-aa62-aa5b19ce808e-metrics-tls\") pod \"dns-default-j54tt\" (UID: \"8b41324a-3369-42ea-aa62-aa5b19ce808e\") " pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:28.188231 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.188208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:28.309919 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.309872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:28.430020 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.429997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:28.479613 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.479594    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j54tt"]
Sep 18 18:45:28.482059 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:28.482035    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b41324a_3369_42ea_aa62_aa5b19ce808e.slice/crio-a18626245792a0a25354a002a5222d0abcde3d0124737a2a117fe4380edc1e34 WatchSource:0}: Error finding container a18626245792a0a25354a002a5222d0abcde3d0124737a2a117fe4380edc1e34: Status 404 returned error can't find the container with id a18626245792a0a25354a002a5222d0abcde3d0124737a2a117fe4380edc1e34
Sep 18 18:45:28.529632 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.529608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:28.678522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.678490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j54tt" event={"ID":"8b41324a-3369-42ea-aa62-aa5b19ce808e","Type":"ContainerStarted","Data":"a18626245792a0a25354a002a5222d0abcde3d0124737a2a117fe4380edc1e34"}
Sep 18 18:45:28.911689 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:28.911668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:29.310637 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:29.310576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:29.413689 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:29.413667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:29.511807 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:29.511782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:29.740149 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:29.740116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:29.911541 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:29.911512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:30.326065 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:30.326031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:30.923709 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:30.923670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:31.023762 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.023698    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:31.241315 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.241295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:31.342165 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.342106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:31.689804 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.689782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j54tt" event={"ID":"8b41324a-3369-42ea-aa62-aa5b19ce808e","Type":"ContainerStarted","Data":"e394ff269b9b0dd72fe6ad5e247d90e477306258a24df51d6e454c35d3b4d672"}
Sep 18 18:45:31.689984 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.689824    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j54tt" event={"ID":"8b41324a-3369-42ea-aa62-aa5b19ce808e","Type":"ContainerStarted","Data":"153e546872530b47303ba75b5402de87166252f49471580e2366e8645a04432e"}
Sep 18 18:45:31.689984 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.689917    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:31.729418 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.729374    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j54tt" podStartSLOduration=17.471446476 podStartE2EDuration="19.729363253s" totalImagesPullingTime="2.257916777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:31.727754493 +0000 UTC m=+55.520431436" watchObservedRunningTime="2026-09-18 18:45:31.729363253 +0000 UTC m=+55.522040181"
Sep 18 18:45:31.745546 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:31.745529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:32.125330 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:32.125263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:32.368970 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:32.368950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:32.724172 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:32.724147    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:32.811754 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:32.811719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:33.040722 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.040679    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:33.137878 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.137856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:33.319171 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.319119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:33.332302 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.332279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:33.332374 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.332315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:33.332438 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:33.332409    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:33.332486 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:33.332462    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls podName:03f46265-978a-4df5-a719-87f818fe09a1 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:49.332447154 +0000 UTC m=+73.125124062 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2twxf" (UID: "03f46265-978a-4df5-a719-87f818fe09a1") : secret "insights-runtime-extractor-tls" not found
Sep 18 18:45:33.332813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.332795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03f46265-978a-4df5-a719-87f818fe09a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:33.499104 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.499070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:33.567573 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.567541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:33.711813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.711783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:33.900785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:33.900746    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:34.112900 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:34.112820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:34.309730 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:34.309713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:34.505682 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:34.505663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:34.914164 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:34.914140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:35.112015 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:35.111993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:35.197756 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:35.197705    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:35.329652 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:35.329626    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:35.552223 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:35.552170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:35.640718 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:35.640700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:35.751574 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:35.751551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:37.616148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:37.616131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5kmhl"
Sep 18 18:45:40.457952 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:40.457920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:41.194221 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:41.194181    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:41.259948 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:41.259921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:41.696136 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:41.696117    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j54tt"
Sep 18 18:45:42.997605 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:42.997583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:42.997909 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:42.997613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:42.997909 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:42.997636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:43.000047 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.000032    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 18:45:43.000647 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.000631    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 18:45:43.000728 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.000640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 18:45:43.010667 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.010646    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 18:45:43.010771 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.010730    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d0f8445-381e-4d78-b730-ac9eb826b144-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqr\" (UID: \"4d0f8445-381e-4d78-b730-ac9eb826b144\") " pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:43.010870 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.010852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129a6d33-4815-4918-9f85-bbe434e773ea-metrics-certs\") pod \"network-metrics-daemon-6xbrr\" (UID: \"129a6d33-4815-4918-9f85-bbe434e773ea\") " pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:43.021775 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.021741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvd5\" (UniqueName: \"kubernetes.io/projected/3fe33d9a-e11a-4bb4-8535-777727b79a84-kube-api-access-mlvd5\") pod \"network-check-target-ghpld\" (UID: \"3fe33d9a-e11a-4bb4-8535-777727b79a84\") " pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:43.244694 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.244677    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdsv4\""
Sep 18 18:45:43.249569 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.249535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqr"
Sep 18 18:45:43.252949 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.252930    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xbrr"
Sep 18 18:45:43.258963 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.258942    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vk59r\""
Sep 18 18:45:43.265841 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.265819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:43.538769 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.538701    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxqr"]
Sep 18 18:45:43.539495 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.539448    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xbrr"]
Sep 18 18:45:43.540438 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.540413    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ghpld"]
Sep 18 18:45:43.541838 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:43.541817    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0f8445_381e_4d78_b730_ac9eb826b144.slice/crio-f38f4d80490820fb086c728238ca2eaa77f2b4fe6e4fe8aa57319e6397e4e94b WatchSource:0}: Error finding container f38f4d80490820fb086c728238ca2eaa77f2b4fe6e4fe8aa57319e6397e4e94b: Status 404 returned error can't find the container with id f38f4d80490820fb086c728238ca2eaa77f2b4fe6e4fe8aa57319e6397e4e94b
Sep 18 18:45:43.542542 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:43.542449    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe33d9a_e11a_4bb4_8535_777727b79a84.slice/crio-77659c1caf3fba704a34abfb3972eea64c6d6133fe5e28d95d28604855f7986f WatchSource:0}: Error finding container 77659c1caf3fba704a34abfb3972eea64c6d6133fe5e28d95d28604855f7986f: Status 404 returned error can't find the container with id 77659c1caf3fba704a34abfb3972eea64c6d6133fe5e28d95d28604855f7986f
Sep 18 18:45:43.543270 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:43.543256    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129a6d33_4815_4918_9f85_bbe434e773ea.slice/crio-cc7f9d2dfd0a8f801eab77427cc010d2d7235dea49fdea5f4cb1b12b6a75bd1f WatchSource:0}: Error finding container cc7f9d2dfd0a8f801eab77427cc010d2d7235dea49fdea5f4cb1b12b6a75bd1f: Status 404 returned error can't find the container with id cc7f9d2dfd0a8f801eab77427cc010d2d7235dea49fdea5f4cb1b12b6a75bd1f
Sep 18 18:45:43.723529 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.721809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xbrr" event={"ID":"129a6d33-4815-4918-9f85-bbe434e773ea","Type":"ContainerStarted","Data":"cc7f9d2dfd0a8f801eab77427cc010d2d7235dea49fdea5f4cb1b12b6a75bd1f"}
Sep 18 18:45:43.724633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.724607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrxqr" event={"ID":"4d0f8445-381e-4d78-b730-ac9eb826b144","Type":"ContainerStarted","Data":"f38f4d80490820fb086c728238ca2eaa77f2b4fe6e4fe8aa57319e6397e4e94b"}
Sep 18 18:45:43.725867 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.725848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ghpld" event={"ID":"3fe33d9a-e11a-4bb4-8535-777727b79a84","Type":"ContainerStarted","Data":"77659c1caf3fba704a34abfb3972eea64c6d6133fe5e28d95d28604855f7986f"}
Sep 18 18:45:43.914834 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.914816    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"]
Sep 18 18:45:43.917421 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.917407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:43.919892 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.919864    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 18:45:43.920033 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.920013    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 18:45:43.920728 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.920712    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 18:45:43.920816 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.920725    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 18:45:43.925863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.925840    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"]
Sep 18 18:45:43.927627 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.927612    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:43.929532 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.929511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 18:45:43.929825 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.929779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8hf7m\""
Sep 18 18:45:43.934176 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.934039    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"]
Sep 18 18:45:43.945801 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:43.945765    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"]
Sep 18 18:45:44.006869 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.006842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpgp\" (UniqueName: \"kubernetes.io/projected/e8075f92-7718-4d9a-8858-122dde2a4d67-kube-api-access-rkpgp\") pod \"managed-serviceaccount-addon-agent-7f68996bf8-kd6rt\" (UID: \"e8075f92-7718-4d9a-8858-122dde2a4d67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:44.007207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.006874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b64272fe-16bf-4bd4-a13b-1bdb601b98be-klusterlet-config\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.007207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.006917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxpf\" (UniqueName: \"kubernetes.io/projected/b64272fe-16bf-4bd4-a13b-1bdb601b98be-kube-api-access-5xxpf\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.007207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.007010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b64272fe-16bf-4bd4-a13b-1bdb601b98be-tmp\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.007207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.007046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8075f92-7718-4d9a-8858-122dde2a4d67-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f68996bf8-kd6rt\" (UID: \"e8075f92-7718-4d9a-8858-122dde2a4d67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:44.095778 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.095697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:44.108905 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.107661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b64272fe-16bf-4bd4-a13b-1bdb601b98be-tmp\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.108905 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.107699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8075f92-7718-4d9a-8858-122dde2a4d67-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f68996bf8-kd6rt\" (UID: \"e8075f92-7718-4d9a-8858-122dde2a4d67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:44.108905 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.107754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpgp\" (UniqueName: \"kubernetes.io/projected/e8075f92-7718-4d9a-8858-122dde2a4d67-kube-api-access-rkpgp\") pod \"managed-serviceaccount-addon-agent-7f68996bf8-kd6rt\" (UID: \"e8075f92-7718-4d9a-8858-122dde2a4d67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:44.108905 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.107778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b64272fe-16bf-4bd4-a13b-1bdb601b98be-klusterlet-config\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.108905 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.107808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxpf\" (UniqueName: \"kubernetes.io/projected/b64272fe-16bf-4bd4-a13b-1bdb601b98be-kube-api-access-5xxpf\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.108905 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.108349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b64272fe-16bf-4bd4-a13b-1bdb601b98be-tmp\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.113267 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.113220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8075f92-7718-4d9a-8858-122dde2a4d67-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f68996bf8-kd6rt\" (UID: \"e8075f92-7718-4d9a-8858-122dde2a4d67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:44.121849 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.121823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxpf\" (UniqueName: \"kubernetes.io/projected/b64272fe-16bf-4bd4-a13b-1bdb601b98be-kube-api-access-5xxpf\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.121976 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.121921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b64272fe-16bf-4bd4-a13b-1bdb601b98be-klusterlet-config\") pod \"klusterlet-addon-workmgr-776fc987fc-znv6h\" (UID: \"b64272fe-16bf-4bd4-a13b-1bdb601b98be\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.128020 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.127994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpgp\" (UniqueName: \"kubernetes.io/projected/e8075f92-7718-4d9a-8858-122dde2a4d67-kube-api-access-rkpgp\") pod \"managed-serviceaccount-addon-agent-7f68996bf8-kd6rt\" (UID: \"e8075f92-7718-4d9a-8858-122dde2a4d67\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:44.183234 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.183163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:44.227013 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.226986    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:44.242973 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.242949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"
Sep 18 18:45:44.291251 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.291143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:44.387452 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.387423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:44.529331 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.529216    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt"]
Sep 18 18:45:44.539051 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.539024    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"]
Sep 18 18:45:44.651186 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:44.651153    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8075f92_7718_4d9a_8858_122dde2a4d67.slice/crio-51e7e4df63ccbb422fcb9931d31b9929fd5ba0c5b9c6253dccad3a972f2f4795 WatchSource:0}: Error finding container 51e7e4df63ccbb422fcb9931d31b9929fd5ba0c5b9c6253dccad3a972f2f4795: Status 404 returned error can't find the container with id 51e7e4df63ccbb422fcb9931d31b9929fd5ba0c5b9c6253dccad3a972f2f4795
Sep 18 18:45:44.652376 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:44.652298    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64272fe_16bf_4bd4_a13b_1bdb601b98be.slice/crio-2554cfc1c26a71446eb4f204f1f7a4ec24a5152fe13324185c81e30b1480917f WatchSource:0}: Error finding container 2554cfc1c26a71446eb4f204f1f7a4ec24a5152fe13324185c81e30b1480917f: Status 404 returned error can't find the container with id 2554cfc1c26a71446eb4f204f1f7a4ec24a5152fe13324185c81e30b1480917f
Sep 18 18:45:44.736048 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.736012    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt" event={"ID":"e8075f92-7718-4d9a-8858-122dde2a4d67","Type":"ContainerStarted","Data":"51e7e4df63ccbb422fcb9931d31b9929fd5ba0c5b9c6253dccad3a972f2f4795"}
Sep 18 18:45:44.737474 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.737438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h" event={"ID":"b64272fe-16bf-4bd4-a13b-1bdb601b98be","Type":"ContainerStarted","Data":"2554cfc1c26a71446eb4f204f1f7a4ec24a5152fe13324185c81e30b1480917f"}
Sep 18 18:45:44.810231 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.810203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:44.914542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:44.914504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:45.048280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.048202    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"]
Sep 18 18:45:45.050862 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.050343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:45.053676 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.053653    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7z74k\""
Sep 18 18:45:45.053926 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.053872    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 18:45:45.067692 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.067637    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"]
Sep 18 18:45:45.076288 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.076263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:45.120865 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.120840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a73b467-7eec-4c57-b74e-9b28b48d9bb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9x7ls\" (UID: \"1a73b467-7eec-4c57-b74e-9b28b48d9bb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:45.221739 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.221714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a73b467-7eec-4c57-b74e-9b28b48d9bb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9x7ls\" (UID: \"1a73b467-7eec-4c57-b74e-9b28b48d9bb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:45.221918 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:45.221902    2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 18:45:45.221986 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:45:45.221973    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a73b467-7eec-4c57-b74e-9b28b48d9bb9-tls-certificates podName:1a73b467-7eec-4c57-b74e-9b28b48d9bb9 nodeName:}" failed. No retries permitted until 2026-09-18 18:45:45.721953048 +0000 UTC m=+69.514629957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1a73b467-7eec-4c57-b74e-9b28b48d9bb9-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-9x7ls" (UID: "1a73b467-7eec-4c57-b74e-9b28b48d9bb9") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 18 18:45:45.256085 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.256058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:45.397770 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.397692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:45.582390 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.582358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:45.652344 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.652262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:45.726271 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.726238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a73b467-7eec-4c57-b74e-9b28b48d9bb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9x7ls\" (UID: \"1a73b467-7eec-4c57-b74e-9b28b48d9bb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:45.738382 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.737973    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1a73b467-7eec-4c57-b74e-9b28b48d9bb9-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9x7ls\" (UID: \"1a73b467-7eec-4c57-b74e-9b28b48d9bb9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:45.757920 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.757443    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xbrr" event={"ID":"129a6d33-4815-4918-9f85-bbe434e773ea","Type":"ContainerStarted","Data":"36ab3ea98528607adcc3ed97887596fc53ea3bb2974ebb30633097dfa08dbdc2"}
Sep 18 18:45:45.757920 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.757550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xbrr" event={"ID":"129a6d33-4815-4918-9f85-bbe434e773ea","Type":"ContainerStarted","Data":"afa0058359c90fee80f80f799500cffdde6911feb561e616b8dc77909ccb2a96"}
Sep 18 18:45:45.776908 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.776188    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6xbrr" podStartSLOduration=66.6424233 podStartE2EDuration="1m7.776171652s" totalImagesPullingTime="1.133748352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:45.775946576 +0000 UTC m=+69.568623525" watchObservedRunningTime="2026-09-18 18:45:45.776171652 +0000 UTC m=+69.568848582"
Sep 18 18:45:45.962395 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:45.962049    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:46.254434 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:46.254357    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"]
Sep 18 18:45:46.579729 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:46.579636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:46.967305 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:46.967276    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:47.053155 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:47.053126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:49.358438 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:49.358059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:49.361904 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:49.360760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03f46265-978a-4df5-a719-87f818fe09a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2twxf\" (UID: \"03f46265-978a-4df5-a719-87f818fe09a1\") " pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:49.639507 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:49.639327    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zff62\""
Sep 18 18:45:49.647475 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:49.647448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2twxf"
Sep 18 18:45:49.733639 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:49.733607    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a73b467_7eec_4c57_b74e_9b28b48d9bb9.slice/crio-f66c3a5e5cfa2be220fe055acdebf860dfaaed8f8b0db82d553ed0fdb86ab8d4 WatchSource:0}: Error finding container f66c3a5e5cfa2be220fe055acdebf860dfaaed8f8b0db82d553ed0fdb86ab8d4: Status 404 returned error can't find the container with id f66c3a5e5cfa2be220fe055acdebf860dfaaed8f8b0db82d553ed0fdb86ab8d4
Sep 18 18:45:49.773508 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:49.773483    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls" event={"ID":"1a73b467-7eec-4c57-b74e-9b28b48d9bb9","Type":"ContainerStarted","Data":"f66c3a5e5cfa2be220fe055acdebf860dfaaed8f8b0db82d553ed0fdb86ab8d4"}
Sep 18 18:45:50.495578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.495446    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2twxf"]
Sep 18 18:45:50.499551 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:50.499518    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f46265_978a_4df5_a719_87f818fe09a1.slice/crio-af3f88e127c7e8345081841b498f4f3f739d597b935f7b9a9a0ca85b4758fb7e WatchSource:0}: Error finding container af3f88e127c7e8345081841b498f4f3f739d597b935f7b9a9a0ca85b4758fb7e: Status 404 returned error can't find the container with id af3f88e127c7e8345081841b498f4f3f739d597b935f7b9a9a0ca85b4758fb7e
Sep 18 18:45:50.780474 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.780391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ghpld" event={"ID":"3fe33d9a-e11a-4bb4-8535-777727b79a84","Type":"ContainerStarted","Data":"971dc346a784f6e4e7a7b8e12465de675285df7e73e0241b1acf78afd7a7764c"}
Sep 18 18:45:50.780633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.780611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:45:50.782550 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.782522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2twxf" event={"ID":"03f46265-978a-4df5-a719-87f818fe09a1","Type":"ContainerStarted","Data":"f5413fdaa204123b4e853a6204f0f6ac069adb274fb25ee58a4ce386c4491392"}
Sep 18 18:45:50.782678 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.782577    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2twxf" event={"ID":"03f46265-978a-4df5-a719-87f818fe09a1","Type":"ContainerStarted","Data":"af3f88e127c7e8345081841b498f4f3f739d597b935f7b9a9a0ca85b4758fb7e"}
Sep 18 18:45:50.784709 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.784370    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrxqr" event={"ID":"4d0f8445-381e-4d78-b730-ac9eb826b144","Type":"ContainerStarted","Data":"8b14041429eb164f567dbde26f25ae3cfb16be63d590ff92aa9a3129c3319924"}
Sep 18 18:45:50.786663 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.786638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt" event={"ID":"e8075f92-7718-4d9a-8858-122dde2a4d67","Type":"ContainerStarted","Data":"f0de90231aa1b5bb99e3666cadec3fcaaf19cdb4732a5bbbb7d2824a6130ed00"}
Sep 18 18:45:50.816565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.816519    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ghpld" podStartSLOduration=66.137174796 podStartE2EDuration="1m12.816500917s" totalImagesPullingTime="6.679326121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:50.816435594 +0000 UTC m=+74.609112548" watchObservedRunningTime="2026-09-18 18:45:50.816500917 +0000 UTC m=+74.609177848"
Sep 18 18:45:50.840339 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.840294    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68996bf8-kd6rt" podStartSLOduration=2.26594094 podStartE2EDuration="7.84027474s" totalImagesPullingTime="5.5743338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:50.840071651 +0000 UTC m=+74.632748581" watchObservedRunningTime="2026-09-18 18:45:50.84027474 +0000 UTC m=+74.632951671"
Sep 18 18:45:50.858774 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:50.858077    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrxqr" podStartSLOduration=66.171094037 podStartE2EDuration="1m12.858060708s" totalImagesPullingTime="6.686966671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:50.857447837 +0000 UTC m=+74.650124791" watchObservedRunningTime="2026-09-18 18:45:50.858060708 +0000 UTC m=+74.650737642"
Sep 18 18:45:51.510535 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.510505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:51.608788 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.608741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:51.812684 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.812611    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b8c55c6d7-q6h67"]
Sep 18 18:45:51.814869 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.814839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.819122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.819096    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 18:45:51.819122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.819108    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 18:45:51.819356 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.819131    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-85nlj\""
Sep 18 18:45:51.819356 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.819160    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 18:45:51.819649 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.819627    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 18:45:51.819649 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.819644    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 18:45:51.819803 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.819693    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 18:45:51.822095 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.821752    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 18:45:51.842729 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.842692    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b8c55c6d7-q6h67"]
Sep 18 18:45:51.878044 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.878010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-config\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.878149 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.878049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-oauth-serving-cert\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.878149 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.878096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-serving-cert\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.878247 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.878148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-service-ca\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.878247 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.878195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-oauth-config\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.878247 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.878226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfhb\" (UniqueName: \"kubernetes.io/projected/c4889c18-2c2a-4501-b53d-d2fddb6419bb-kube-api-access-vkfhb\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.894292 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.894260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:51.979059 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-config\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-oauth-serving-cert\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-serving-cert\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-service-ca\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-oauth-config\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979401 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfhb\" (UniqueName: \"kubernetes.io/projected/c4889c18-2c2a-4501-b53d-d2fddb6419bb-kube-api-access-vkfhb\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979848 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-config\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979955 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-service-ca\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.979955 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.979824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-oauth-serving-cert\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.982032 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.982011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-serving-cert\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.982124 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.982065    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-oauth-config\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:51.987802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:51.987774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfhb\" (UniqueName: \"kubernetes.io/projected/c4889c18-2c2a-4501-b53d-d2fddb6419bb-kube-api-access-vkfhb\") pod \"console-7b8c55c6d7-q6h67\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") " pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:52.004979 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:52.004953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:52.124871 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:52.124799    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:45:52.347643 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:52.347605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:52.424366 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:52.424337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:52.589391 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:52.589354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:53.052313 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.052196    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b8c55c6d7-q6h67"]
Sep 18 18:45:53.054641 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:53.054616    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4889c18_2c2a_4501_b53d_d2fddb6419bb.slice/crio-1d6a514b82fc568447fbf09f6c6e53f8b8dc997c60dde0191fabe05f0604a3bf WatchSource:0}: Error finding container 1d6a514b82fc568447fbf09f6c6e53f8b8dc997c60dde0191fabe05f0604a3bf: Status 404 returned error can't find the container with id 1d6a514b82fc568447fbf09f6c6e53f8b8dc997c60dde0191fabe05f0604a3bf
Sep 18 18:45:53.801176 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.801136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls" event={"ID":"1a73b467-7eec-4c57-b74e-9b28b48d9bb9","Type":"ContainerStarted","Data":"1bdcdf6e280e348e017f5256164029b634935f6ff67abc9d5765733f9fdc5259"}
Sep 18 18:45:53.801948 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.801918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:53.809214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.809026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2twxf" event={"ID":"03f46265-978a-4df5-a719-87f818fe09a1","Type":"ContainerStarted","Data":"a3559d9a1ef7907eb951a5c5ffde1b9728261706fd526509605e56cecbfadae8"}
Sep 18 18:45:53.809214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.809173    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls"
Sep 18 18:45:53.813021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.812269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h" event={"ID":"b64272fe-16bf-4bd4-a13b-1bdb601b98be","Type":"ContainerStarted","Data":"084afc16ebe1860a0120a5d55be1b229df9de9b19ac75dc70f3022bb3965f1c5"}
Sep 18 18:45:53.813021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.812965    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:53.814390 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.814372    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h"
Sep 18 18:45:53.816492 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.816467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8c55c6d7-q6h67" event={"ID":"c4889c18-2c2a-4501-b53d-d2fddb6419bb","Type":"ContainerStarted","Data":"1d6a514b82fc568447fbf09f6c6e53f8b8dc997c60dde0191fabe05f0604a3bf"}
Sep 18 18:45:53.908034 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.907980    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-776fc987fc-znv6h" podStartSLOduration=2.652268889 podStartE2EDuration="10.907961543s" totalImagesPullingTime="8.255692654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:53.90753671 +0000 UTC m=+77.700213640" watchObservedRunningTime="2026-09-18 18:45:53.907961543 +0000 UTC m=+77.700638474"
Sep 18 18:45:53.909280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:53.909237    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9x7ls" podStartSLOduration=5.990964518 podStartE2EDuration="8.909223659s" totalImagesPullingTime="2.918259141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:53.845157219 +0000 UTC m=+77.637834146" watchObservedRunningTime="2026-09-18 18:45:53.909223659 +0000 UTC m=+77.701900590"
Sep 18 18:45:54.076490 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:54.076402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:54.164235 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:54.164200    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:54.613064 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:54.613020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:54.783369 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:54.783296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:55.716671 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.716637    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7758489dd-jvpdj"]
Sep 18 18:45:55.723357 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.723335    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.732353 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.732334    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 18:45:55.736794 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.736771    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7758489dd-jvpdj"]
Sep 18 18:45:55.808309 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.808275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-oauth-serving-cert\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.808309 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.808318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-serving-cert\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.808542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.808344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-trusted-ca-bundle\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.808542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.808438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-console-config\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.808542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.808480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjkb\" (UniqueName: \"kubernetes.io/projected/53ddc901-6593-4292-9088-19ab86c9701d-kube-api-access-rtjkb\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.808542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.808522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-service-ca\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.808694 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.808557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-oauth-config\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.909649 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.909618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-oauth-config\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.909804 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.909685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-oauth-serving-cert\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.909804 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.909709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-serving-cert\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.909804 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.909735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-trusted-ca-bundle\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.910547 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.910518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-console-config\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.910672 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.910590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjkb\" (UniqueName: \"kubernetes.io/projected/53ddc901-6593-4292-9088-19ab86c9701d-kube-api-access-rtjkb\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.910672 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.910593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-oauth-serving-cert\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.910672 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.910631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-service-ca\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.911038 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.910983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-trusted-ca-bundle\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.911497 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.911471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-console-config\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.911603 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.911480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-service-ca\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.913084 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.913061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-oauth-config\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.913690 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.913662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-serving-cert\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:55.923741 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:55.923714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjkb\" (UniqueName: \"kubernetes.io/projected/53ddc901-6593-4292-9088-19ab86c9701d-kube-api-access-rtjkb\") pod \"console-7758489dd-jvpdj\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") " pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:56.035097 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:56.035020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:45:56.346253 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:56.346182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:56.448851 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:56.448810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:45:57.513175 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.513097    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7758489dd-jvpdj"]
Sep 18 18:45:57.515514 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:45:57.515486    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ddc901_6593_4292_9088_19ab86c9701d.slice/crio-143c7e8bbbea5dc691678e69410d416384454af59e3a6dcb5af3babd828aa764 WatchSource:0}: Error finding container 143c7e8bbbea5dc691678e69410d416384454af59e3a6dcb5af3babd828aa764: Status 404 returned error can't find the container with id 143c7e8bbbea5dc691678e69410d416384454af59e3a6dcb5af3babd828aa764
Sep 18 18:45:57.833932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.833859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2twxf" event={"ID":"03f46265-978a-4df5-a719-87f818fe09a1","Type":"ContainerStarted","Data":"924eb5c88cfdf361fbd971a38f73642f9a61e16d8ae5a313a5edd29d6d50c741"}
Sep 18 18:45:57.835546 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.835518    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8c55c6d7-q6h67" event={"ID":"c4889c18-2c2a-4501-b53d-d2fddb6419bb","Type":"ContainerStarted","Data":"7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b"}
Sep 18 18:45:57.836982 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.836959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758489dd-jvpdj" event={"ID":"53ddc901-6593-4292-9088-19ab86c9701d","Type":"ContainerStarted","Data":"deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86"}
Sep 18 18:45:57.837073 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.837002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758489dd-jvpdj" event={"ID":"53ddc901-6593-4292-9088-19ab86c9701d","Type":"ContainerStarted","Data":"143c7e8bbbea5dc691678e69410d416384454af59e3a6dcb5af3babd828aa764"}
Sep 18 18:45:57.854113 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.854071    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2twxf" podStartSLOduration=34.227969092 podStartE2EDuration="40.854057172s" totalImagesPullingTime="6.62608808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 18:45:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:57.853563285 +0000 UTC m=+81.646240234" watchObservedRunningTime="2026-09-18 18:45:57.854057172 +0000 UTC m=+81.646734102"
Sep 18 18:45:57.878973 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.878934    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b8c55c6d7-q6h67" podStartSLOduration=2.626362711 podStartE2EDuration="6.878921955s" totalImagesPullingTime="4.252559244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:57.878589025 +0000 UTC m=+81.671265956" watchObservedRunningTime="2026-09-18 18:45:57.878921955 +0000 UTC m=+81.671598929"
Sep 18 18:45:57.899601 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:45:57.899559    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7758489dd-jvpdj" podStartSLOduration=2.899547995 podStartE2EDuration="2.899547995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:45:57.899184436 +0000 UTC m=+81.691861392" watchObservedRunningTime="2026-09-18 18:45:57.899547995 +0000 UTC m=+81.692224924"
Sep 18 18:46:00.415833 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:00.415805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:00.520462 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:00.520431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:00.618289 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:00.618267    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:01.220619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:01.220591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:01.297321 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:01.297281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:02.014993 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:02.014950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:02.110845 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:02.110808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:02.124901 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:02.124868    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:46:02.124979 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:02.124953    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:46:02.129099 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:02.129083    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:46:02.631803 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:02.631762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:02.858385 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:02.858364    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:46:03.061256 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:03.061232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:03.167387 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:03.167341    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:05.259874 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.259839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:05.512105 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.512037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:05.670246 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.670217    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zf9vv"]
Sep 18 18:46:05.675333 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.675313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.678323 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.678301    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 18:46:05.678440 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.678383    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 18:46:05.678498 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.678469    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 18:46:05.678549 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.678511    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 18:46:05.679318 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.679298    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 18:46:05.679318 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.679309    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 18:46:05.679473 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.679356    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-js7p5\""
Sep 18 18:46:05.685785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.685766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:05.778408 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778362    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/69c2029f-499c-4361-bc76-869618abd272-node-exporter-textfile\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778408 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/69c2029f-499c-4361-bc76-869618abd272-node-exporter-tls\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778559 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-sys\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778559 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69c2029f-499c-4361-bc76-869618abd272-metrics-client-ca\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778559 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69c2029f-499c-4361-bc76-869618abd272-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778559 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx4bq\" (UniqueName: \"kubernetes.io/projected/69c2029f-499c-4361-bc76-869618abd272-kube-api-access-wx4bq\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778742 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-root\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778742 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-node-exporter-wtmp\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.778742 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.778658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/69c2029f-499c-4361-bc76-869618abd272-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.780445 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.780420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:05.879358 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69c2029f-499c-4361-bc76-869618abd272-metrics-client-ca\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879447 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69c2029f-499c-4361-bc76-869618abd272-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879447 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx4bq\" (UniqueName: \"kubernetes.io/projected/69c2029f-499c-4361-bc76-869618abd272-kube-api-access-wx4bq\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879447 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-root\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879447 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-node-exporter-wtmp\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879447 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/69c2029f-499c-4361-bc76-869618abd272-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-root\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/69c2029f-499c-4361-bc76-869618abd272-node-exporter-textfile\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/69c2029f-499c-4361-bc76-869618abd272-node-exporter-tls\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-node-exporter-wtmp\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-sys\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c2029f-499c-4361-bc76-869618abd272-sys\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.879922 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.879814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/69c2029f-499c-4361-bc76-869618abd272-node-exporter-textfile\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.880035 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.880011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/69c2029f-499c-4361-bc76-869618abd272-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.880102 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.880014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69c2029f-499c-4361-bc76-869618abd272-metrics-client-ca\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.881821 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.881802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/69c2029f-499c-4361-bc76-869618abd272-node-exporter-tls\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.881980 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.881964    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69c2029f-499c-4361-bc76-869618abd272-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.887748 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.887729    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx4bq\" (UniqueName: \"kubernetes.io/projected/69c2029f-499c-4361-bc76-869618abd272-kube-api-access-wx4bq\") pod \"node-exporter-zf9vv\" (UID: \"69c2029f-499c-4361-bc76-869618abd272\") " pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.888362 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.888343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:05.985993 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.985974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf9vv"
Sep 18 18:46:05.995380 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:05.995354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:05.997416 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:46:05.997388    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c2029f_499c_4361_bc76_869618abd272.slice/crio-1431d85ed1a3320f467ae5f35f6313a8a7a2a4aaa3d23bb2c12d9d3dd7c68105 WatchSource:0}: Error finding container 1431d85ed1a3320f467ae5f35f6313a8a7a2a4aaa3d23bb2c12d9d3dd7c68105: Status 404 returned error can't find the container with id 1431d85ed1a3320f467ae5f35f6313a8a7a2a4aaa3d23bb2c12d9d3dd7c68105
Sep 18 18:46:06.036156 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.036102    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:46:06.036243 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.036164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:46:06.040449 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.040433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:46:06.093384 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.093362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:06.546074 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.546048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:06.668760 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.668730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:06.709065 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.709041    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:46:06.712598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.712580    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.714983 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.714956    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-66gnc\""
Sep 18 18:46:06.717411 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717389    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 18:46:06.717561 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717421    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 18:46:06.717561 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717429    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 18:46:06.717663 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717435    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 18:46:06.717663 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717626    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 18:46:06.717810 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717668    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 18:46:06.717810 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717655    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 18:46:06.717810 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717668    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 18:46:06.718058 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.717926    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 18:46:06.736368 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.736345    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:46:06.784441 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784546 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-web-config\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784546 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784546 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-config-volume\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784546 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784726 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784726 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784726 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784726 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784726 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.784726 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2lg\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-kube-api-access-vf2lg\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.785029 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-config-out\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.785029 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.784795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.868034 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.867971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:06.868543 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.868520    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9vv" event={"ID":"69c2029f-499c-4361-bc76-869618abd272","Type":"ContainerStarted","Data":"1431d85ed1a3320f467ae5f35f6313a8a7a2a4aaa3d23bb2c12d9d3dd7c68105"}
Sep 18 18:46:06.873041 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.873023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:46:06.885535 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885635 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885635 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2lg\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-kube-api-access-vf2lg\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885635 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885603    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-config-out\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885635 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-web-config\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-config-volume\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.885828 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.886172 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.885852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.886706 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.886682    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.886796 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.886775    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.887181 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.887157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.890036 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.889467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.890036 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.889863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.890784 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.890740    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-web-config\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.890870 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.890784    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.890870 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.890794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.891176 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.891157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-config-volume\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.891582 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.891561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.891654 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.891580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.892273 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.892251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-config-out\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.897204 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.897181    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2lg\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-kube-api-access-vf2lg\") pod \"alertmanager-main-0\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:06.966982 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:06.966951    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b8c55c6d7-q6h67"]
Sep 18 18:46:07.023652 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.023625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:46:07.029858 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.029830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:07.147877 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.147852    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:07.253939 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.253909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:07.420838 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.420813    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:46:07.424234 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:46:07.424206    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecce41ba_6178_4be2_955d_2ad715e93edd.slice/crio-900a4c72c63822e05d5c8c0fabe6864b2596d1f0a75ed3aa54e719d44b5d0e9f WatchSource:0}: Error finding container 900a4c72c63822e05d5c8c0fabe6864b2596d1f0a75ed3aa54e719d44b5d0e9f: Status 404 returned error can't find the container with id 900a4c72c63822e05d5c8c0fabe6864b2596d1f0a75ed3aa54e719d44b5d0e9f
Sep 18 18:46:07.634053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.634032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:07.734392 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.734342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:07.872738 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.872711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerStarted","Data":"900a4c72c63822e05d5c8c0fabe6864b2596d1f0a75ed3aa54e719d44b5d0e9f"}
Sep 18 18:46:07.874277 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.874259    2582 generic.go:356] "Generic (PLEG): container finished" podID="69c2029f-499c-4361-bc76-869618abd272" containerID="65256af08ea0d65fa6c2f2c8483fb689eb16857d6712c8d7dc98d43870e3e9ed" exitCode=0
Sep 18 18:46:07.874361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:07.874340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9vv" event={"ID":"69c2029f-499c-4361-bc76-869618abd272","Type":"ContainerDied","Data":"65256af08ea0d65fa6c2f2c8483fb689eb16857d6712c8d7dc98d43870e3e9ed"}
Sep 18 18:46:08.441909 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.440522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:08.522220 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.522188    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:08.617620 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.617596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:08.641539 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.641517    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"]
Sep 18 18:46:08.645684 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.645667    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.648148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.648006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 18:46:08.648148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.648024    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 18:46:08.648148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.648024    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7od9hpgsdjlar\""
Sep 18 18:46:08.648148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.648035    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 18:46:08.648148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.648072    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 18:46:08.648148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.648082    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2fgfh\""
Sep 18 18:46:08.648148 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.648024    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 18:46:08.656388 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.656346    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"]
Sep 18 18:46:08.692768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.692708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:08.701617 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701596    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.701699 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701633    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.701699 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.701813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-grpc-tls\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.701813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-tls\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.701932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8532550-35be-4ae9-b209-81f6b19eee69-metrics-client-ca\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.701932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.702019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.701980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjvj\" (UniqueName: \"kubernetes.io/projected/c8532550-35be-4ae9-b209-81f6b19eee69-kube-api-access-mkjvj\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.756333 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.756313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:08.802528 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-tls\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.802611 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8532550-35be-4ae9-b209-81f6b19eee69-metrics-client-ca\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.802611 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.802611 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjvj\" (UniqueName: \"kubernetes.io/projected/c8532550-35be-4ae9-b209-81f6b19eee69-kube-api-access-mkjvj\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.802741 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.802791 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.802791 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.802920 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.802792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-grpc-tls\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.803280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.803254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8532550-35be-4ae9-b209-81f6b19eee69-metrics-client-ca\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.805364 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.805336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.805426 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.805394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.805739 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.805721    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-grpc-tls\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.805822 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.805803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.805870 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.805822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.805943 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.805870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c8532550-35be-4ae9-b209-81f6b19eee69-secret-thanos-querier-tls\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.810285 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.810265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjvj\" (UniqueName: \"kubernetes.io/projected/c8532550-35be-4ae9-b209-81f6b19eee69-kube-api-access-mkjvj\") pod \"thanos-querier-9d8fd8fcf-v69z9\" (UID: \"c8532550-35be-4ae9-b209-81f6b19eee69\") " pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:08.880346 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.880317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9vv" event={"ID":"69c2029f-499c-4361-bc76-869618abd272","Type":"ContainerStarted","Data":"afa8cd82bd93bf290e7124903bf653d6ff25536b9d1d5658d7072f9563a1d033"}
Sep 18 18:46:08.880434 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.880372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf9vv" event={"ID":"69c2029f-499c-4361-bc76-869618abd272","Type":"ContainerStarted","Data":"e679b3b1d01943b53f47c5f24243a781752b047216f41b2e7948623b1fa965dc"}
Sep 18 18:46:08.881924 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.881901    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerID="98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c" exitCode=0
Sep 18 18:46:08.881996 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.881930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c"}
Sep 18 18:46:08.901016 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:08.900978    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zf9vv" podStartSLOduration=2.648585464 podStartE2EDuration="3.90096579s" totalImagesPullingTime="1.187075878s" totalInitContainerRuntime="65.304448ms" isStatefulPod=true podCreationTimestamp="2026-09-18 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:46:08.900222904 +0000 UTC m=+92.692899856" watchObservedRunningTime="2026-09-18 18:46:08.90096579 +0000 UTC m=+92.693642719"
Sep 18 18:46:09.002323 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:09.002274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:09.102193 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:09.102161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:09.281323 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:09.281266    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"]
Sep 18 18:46:09.283716 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:46:09.283691    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8532550_35be_4ae9_b209_81f6b19eee69.slice/crio-754420d49e50f0b1b93d7a8522706dd663c1f7885c2d1b2a2ca9de5d1315992d WatchSource:0}: Error finding container 754420d49e50f0b1b93d7a8522706dd663c1f7885c2d1b2a2ca9de5d1315992d: Status 404 returned error can't find the container with id 754420d49e50f0b1b93d7a8522706dd663c1f7885c2d1b2a2ca9de5d1315992d
Sep 18 18:46:09.887696 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:09.887658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" event={"ID":"c8532550-35be-4ae9-b209-81f6b19eee69","Type":"ContainerStarted","Data":"754420d49e50f0b1b93d7a8522706dd663c1f7885c2d1b2a2ca9de5d1315992d"}
Sep 18 18:46:10.275472 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.275446    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59c9dbf984-nnk5d"]
Sep 18 18:46:10.279035 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.279009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.302256 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.302228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:10.302589 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.302565    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c9dbf984-nnk5d"]
Sep 18 18:46:10.414242 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.414216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:10.417183 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.417155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-service-ca\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.417283 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.417191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-oauth-serving-cert\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.417283 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.417251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-trusted-ca-bundle\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.417283 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.417281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-oauth-config\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.417424 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.417321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-console-config\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.417424 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.417361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdjm\" (UniqueName: \"kubernetes.io/projected/d4f722a7-b8c9-4576-818d-eba1840cea42-kube-api-access-tcdjm\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.417424 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.417389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-serving-cert\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.501580 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.501558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:10.518896 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.518720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-oauth-config\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.518896 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.518776    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-console-config\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.518896 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.518864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdjm\" (UniqueName: \"kubernetes.io/projected/d4f722a7-b8c9-4576-818d-eba1840cea42-kube-api-access-tcdjm\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.519116 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.518909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-serving-cert\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.519116 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.518943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-service-ca\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.519116 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.518967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-oauth-serving-cert\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.519116 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.519018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-trusted-ca-bundle\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.519663 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.519636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-console-config\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.519815 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.519795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-service-ca\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.520062 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.519993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-trusted-ca-bundle\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.520265 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.520244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-oauth-serving-cert\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.521939 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.521826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-serving-cert\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.522286 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.522263    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-oauth-config\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.531707 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.531656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdjm\" (UniqueName: \"kubernetes.io/projected/d4f722a7-b8c9-4576-818d-eba1840cea42-kube-api-access-tcdjm\") pod \"console-59c9dbf984-nnk5d\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") " pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.605187 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.605164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:10.886065 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.885794    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c9dbf984-nnk5d"]
Sep 18 18:46:10.888397 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:46:10.888367    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f722a7_b8c9_4576_818d_eba1840cea42.slice/crio-3a9e0179226034c00901b16962de3994b044a6b2b5545d2e63093c70e3ee8749 WatchSource:0}: Error finding container 3a9e0179226034c00901b16962de3994b044a6b2b5545d2e63093c70e3ee8749: Status 404 returned error can't find the container with id 3a9e0179226034c00901b16962de3994b044a6b2b5545d2e63093c70e3ee8749
Sep 18 18:46:10.895578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.895547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerStarted","Data":"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"}
Sep 18 18:46:10.895675 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.895635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerStarted","Data":"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"}
Sep 18 18:46:10.895738 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.895694    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerStarted","Data":"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"}
Sep 18 18:46:10.895738 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.895726    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerStarted","Data":"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"}
Sep 18 18:46:10.895818 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:10.895753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerStarted","Data":"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"}
Sep 18 18:46:11.901277 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:11.901251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9dbf984-nnk5d" event={"ID":"d4f722a7-b8c9-4576-818d-eba1840cea42","Type":"ContainerStarted","Data":"353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac"}
Sep 18 18:46:11.901604 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:11.901326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9dbf984-nnk5d" event={"ID":"d4f722a7-b8c9-4576-818d-eba1840cea42","Type":"ContainerStarted","Data":"3a9e0179226034c00901b16962de3994b044a6b2b5545d2e63093c70e3ee8749"}
Sep 18 18:46:11.903422 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:11.903401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" event={"ID":"c8532550-35be-4ae9-b209-81f6b19eee69","Type":"ContainerStarted","Data":"5ea38390b9f80757de2019ccff43e69d3d32ad31863ff62ea310215ee5e30bf2"}
Sep 18 18:46:11.903489 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:11.903451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" event={"ID":"c8532550-35be-4ae9-b209-81f6b19eee69","Type":"ContainerStarted","Data":"79a74d4a365d5813371340db1cb23cc71a2ad0cca5e93cdb490f312be0ddd30d"}
Sep 18 18:46:11.903489 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:11.903476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" event={"ID":"c8532550-35be-4ae9-b209-81f6b19eee69","Type":"ContainerStarted","Data":"f6e813a4339ceccef49de4bf9570e108380cf3a2a5e7f005c9a92c328434ec83"}
Sep 18 18:46:11.923120 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:11.922968    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59c9dbf984-nnk5d" podStartSLOduration=1.92295099 podStartE2EDuration="1.92295099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:46:11.922164743 +0000 UTC m=+95.714841672" watchObservedRunningTime="2026-09-18 18:46:11.92295099 +0000 UTC m=+95.715627919"
Sep 18 18:46:11.960354 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:11.960330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:12.909409 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:12.909376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" event={"ID":"c8532550-35be-4ae9-b209-81f6b19eee69","Type":"ContainerStarted","Data":"73a0be18461c4652487614eb9c3acd61b7b9d98f7fe52e72dfc46d142dc43d8d"}
Sep 18 18:46:12.909790 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:12.909451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" event={"ID":"c8532550-35be-4ae9-b209-81f6b19eee69","Type":"ContainerStarted","Data":"4599eeaa99e3abf6ebe9b3b830bdd95d78d4e8f1fd18ab0b0348e3d8173da97f"}
Sep 18 18:46:12.909790 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:12.909574    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:12.909790 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:12.909619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" event={"ID":"c8532550-35be-4ae9-b209-81f6b19eee69","Type":"ContainerStarted","Data":"f9a0d0f3af21e4b0224ac490a60509f2d6706c0acb593e7c5cf4b665d77a8589"}
Sep 18 18:46:12.912494 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:12.912471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerStarted","Data":"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"}
Sep 18 18:46:12.952388 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:12.952337    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9" podStartSLOduration=2.347480123 podStartE2EDuration="4.952319454s" totalImagesPullingTime="2.604839331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:46:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:46:12.951495544 +0000 UTC m=+96.744172504" watchObservedRunningTime="2026-09-18 18:46:12.952319454 +0000 UTC m=+96.744996394"
Sep 18 18:46:12.991648 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:12.991604    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.21912454 podStartE2EDuration="6.991587718s" totalImagesPullingTime="3.756413075s" totalInitContainerRuntime="16.050103ms" isStatefulPod=false podCreationTimestamp="2026-09-18 18:46:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:46:12.990704753 +0000 UTC m=+96.783381702" watchObservedRunningTime="2026-09-18 18:46:12.991587718 +0000 UTC m=+96.784264648"
Sep 18 18:46:14.143968 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:14.143932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:14.214427 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:14.214401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:18.922134 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:18.922107    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9d8fd8fcf-v69z9"
Sep 18 18:46:20.606522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:20.606492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:20.606878 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:20.606539    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:20.610675 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:20.610653    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:20.944516 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:20.944489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:46:20.999604 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:20.999585    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7758489dd-jvpdj"]
Sep 18 18:46:21.128974 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:21.128947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:21.228303 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:21.228247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:21.354372 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:21.354328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:21.793085 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:21.793069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ghpld"
Sep 18 18:46:25.635436 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:25.635401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:25.738558 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:25.738533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:31.987601 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:31.987543    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b8c55c6d7-q6h67" podUID="c4889c18-2c2a-4501-b53d-d2fddb6419bb" containerName="console" containerID="cri-o://7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b" gracePeriod=15
Sep 18 18:46:32.224999 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.224977    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:46:32.363870 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.363808    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-config\") pod \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") "
Sep 18 18:46:32.363870 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.363848    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkfhb\" (UniqueName: \"kubernetes.io/projected/c4889c18-2c2a-4501-b53d-d2fddb6419bb-kube-api-access-vkfhb\") pod \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") "
Sep 18 18:46:32.364038 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.363876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-service-ca\") pod \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") "
Sep 18 18:46:32.364038 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.363936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-oauth-serving-cert\") pod \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") "
Sep 18 18:46:32.364038 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.363974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-serving-cert\") pod \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") "
Sep 18 18:46:32.364038 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.364012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-oauth-config\") pod \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\" (UID: \"c4889c18-2c2a-4501-b53d-d2fddb6419bb\") "
Sep 18 18:46:32.364339 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.364313    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-config" (OuterVolumeSpecName: "console-config") pod "c4889c18-2c2a-4501-b53d-d2fddb6419bb" (UID: "c4889c18-2c2a-4501-b53d-d2fddb6419bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:32.364339 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.364325    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "c4889c18-2c2a-4501-b53d-d2fddb6419bb" (UID: "c4889c18-2c2a-4501-b53d-d2fddb6419bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:32.364432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.364329    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c4889c18-2c2a-4501-b53d-d2fddb6419bb" (UID: "c4889c18-2c2a-4501-b53d-d2fddb6419bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:32.366145 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.366116    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c4889c18-2c2a-4501-b53d-d2fddb6419bb" (UID: "c4889c18-2c2a-4501-b53d-d2fddb6419bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:46:32.366218 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.366166    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4889c18-2c2a-4501-b53d-d2fddb6419bb-kube-api-access-vkfhb" (OuterVolumeSpecName: "kube-api-access-vkfhb") pod "c4889c18-2c2a-4501-b53d-d2fddb6419bb" (UID: "c4889c18-2c2a-4501-b53d-d2fddb6419bb"). InnerVolumeSpecName "kube-api-access-vkfhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:46:32.366279 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.366222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c4889c18-2c2a-4501-b53d-d2fddb6419bb" (UID: "c4889c18-2c2a-4501-b53d-d2fddb6419bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:46:32.464723 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.464696    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:32.464723 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.464720    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkfhb\" (UniqueName: \"kubernetes.io/projected/c4889c18-2c2a-4501-b53d-d2fddb6419bb-kube-api-access-vkfhb\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:32.464723 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.464729    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:32.464863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.464738    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4889c18-2c2a-4501-b53d-d2fddb6419bb-oauth-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:32.464863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.464746    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:32.464863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.464753    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4889c18-2c2a-4501-b53d-d2fddb6419bb-console-oauth-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:32.983276 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.983248    2582 generic.go:356] "Generic (PLEG): container finished" podID="c4889c18-2c2a-4501-b53d-d2fddb6419bb" containerID="7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b" exitCode=0
Sep 18 18:46:32.983362 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.983288    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b8c55c6d7-q6h67"
Sep 18 18:46:32.983362 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.983323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8c55c6d7-q6h67" event={"ID":"c4889c18-2c2a-4501-b53d-d2fddb6419bb","Type":"ContainerDied","Data":"7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b"}
Sep 18 18:46:32.983432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.983363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b8c55c6d7-q6h67" event={"ID":"c4889c18-2c2a-4501-b53d-d2fddb6419bb","Type":"ContainerDied","Data":"1d6a514b82fc568447fbf09f6c6e53f8b8dc997c60dde0191fabe05f0604a3bf"}
Sep 18 18:46:32.983432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.983384    2582 scope.go:117] "RemoveContainer" containerID="7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b"
Sep 18 18:46:32.991938 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.991849    2582 scope.go:117] "RemoveContainer" containerID="7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b"
Sep 18 18:46:32.992143 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:46:32.992125    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b\": container with ID starting with 7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b not found: ID does not exist" containerID="7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b"
Sep 18 18:46:32.992186 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:32.992150    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b"} err="failed to get container status \"7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b\": rpc error: code = NotFound desc = could not find container \"7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b\": container with ID starting with 7629adb20fd637b55a522a3c5d1fa8e4aede795486305de6351a073ed2f9341b not found: ID does not exist"
Sep 18 18:46:33.004653 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:33.004631    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b8c55c6d7-q6h67"]
Sep 18 18:46:33.015956 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:33.015936    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b8c55c6d7-q6h67"]
Sep 18 18:46:34.334056 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:34.334020    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4889c18-2c2a-4501-b53d-d2fddb6419bb" path="/var/lib/kubelet/pods/c4889c18-2c2a-4501-b53d-d2fddb6419bb/volumes"
Sep 18 18:46:36.019246 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:36.019208    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:36.124815 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:36.124784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:38.279037 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:38.279014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:38.395762 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:38.395738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:38.505927 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:38.505867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:46.017676 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.017640    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7758489dd-jvpdj" podUID="53ddc901-6593-4292-9088-19ab86c9701d" containerName="console" containerID="cri-o://deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86" gracePeriod=15
Sep 18 18:46:46.214268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.214240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:46.251678 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.251659    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:46:46.314190 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.314118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:46:46.358022 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358005    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-serving-cert\") pod \"53ddc901-6593-4292-9088-19ab86c9701d\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") "
Sep 18 18:46:46.358125 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-trusted-ca-bundle\") pod \"53ddc901-6593-4292-9088-19ab86c9701d\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") "
Sep 18 18:46:46.358125 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358055    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-console-config\") pod \"53ddc901-6593-4292-9088-19ab86c9701d\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") "
Sep 18 18:46:46.358125 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtjkb\" (UniqueName: \"kubernetes.io/projected/53ddc901-6593-4292-9088-19ab86c9701d-kube-api-access-rtjkb\") pod \"53ddc901-6593-4292-9088-19ab86c9701d\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") "
Sep 18 18:46:46.358125 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358102    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-service-ca\") pod \"53ddc901-6593-4292-9088-19ab86c9701d\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") "
Sep 18 18:46:46.358299 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358181    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-oauth-config\") pod \"53ddc901-6593-4292-9088-19ab86c9701d\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") "
Sep 18 18:46:46.358299 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-oauth-serving-cert\") pod \"53ddc901-6593-4292-9088-19ab86c9701d\" (UID: \"53ddc901-6593-4292-9088-19ab86c9701d\") "
Sep 18 18:46:46.358488 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-console-config" (OuterVolumeSpecName: "console-config") pod "53ddc901-6593-4292-9088-19ab86c9701d" (UID: "53ddc901-6593-4292-9088-19ab86c9701d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:46.358488 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358461    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-service-ca" (OuterVolumeSpecName: "service-ca") pod "53ddc901-6593-4292-9088-19ab86c9701d" (UID: "53ddc901-6593-4292-9088-19ab86c9701d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:46.358861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358483    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "53ddc901-6593-4292-9088-19ab86c9701d" (UID: "53ddc901-6593-4292-9088-19ab86c9701d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:46.358861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.358741    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "53ddc901-6593-4292-9088-19ab86c9701d" (UID: "53ddc901-6593-4292-9088-19ab86c9701d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:46:46.360321 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.360292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "53ddc901-6593-4292-9088-19ab86c9701d" (UID: "53ddc901-6593-4292-9088-19ab86c9701d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:46:46.360321 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.360302    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ddc901-6593-4292-9088-19ab86c9701d-kube-api-access-rtjkb" (OuterVolumeSpecName: "kube-api-access-rtjkb") pod "53ddc901-6593-4292-9088-19ab86c9701d" (UID: "53ddc901-6593-4292-9088-19ab86c9701d"). InnerVolumeSpecName "kube-api-access-rtjkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:46:46.360321 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.360304    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "53ddc901-6593-4292-9088-19ab86c9701d" (UID: "53ddc901-6593-4292-9088-19ab86c9701d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:46:46.458715 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.458695    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:46.458715 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.458716    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-oauth-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:46.458831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.458725    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-oauth-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:46.458831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.458733    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53ddc901-6593-4292-9088-19ab86c9701d-console-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:46.458831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.458748    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-trusted-ca-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:46.458831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.458755    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53ddc901-6593-4292-9088-19ab86c9701d-console-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:46.458831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:46.458763    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtjkb\" (UniqueName: \"kubernetes.io/projected/53ddc901-6593-4292-9088-19ab86c9701d-kube-api-access-rtjkb\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:46:47.031647 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.031619    2582 generic.go:356] "Generic (PLEG): container finished" podID="53ddc901-6593-4292-9088-19ab86c9701d" containerID="deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86" exitCode=0
Sep 18 18:46:47.032025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.031665    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7758489dd-jvpdj"
Sep 18 18:46:47.032025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.031708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758489dd-jvpdj" event={"ID":"53ddc901-6593-4292-9088-19ab86c9701d","Type":"ContainerDied","Data":"deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86"}
Sep 18 18:46:47.032025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.031769    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7758489dd-jvpdj" event={"ID":"53ddc901-6593-4292-9088-19ab86c9701d","Type":"ContainerDied","Data":"143c7e8bbbea5dc691678e69410d416384454af59e3a6dcb5af3babd828aa764"}
Sep 18 18:46:47.032025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.031798    2582 scope.go:117] "RemoveContainer" containerID="deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86"
Sep 18 18:46:47.041185 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.041171    2582 scope.go:117] "RemoveContainer" containerID="deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86"
Sep 18 18:46:47.041473 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:46:47.041448    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86\": container with ID starting with deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86 not found: ID does not exist" containerID="deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86"
Sep 18 18:46:47.041542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.041484    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86"} err="failed to get container status \"deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86\": rpc error: code = NotFound desc = could not find container \"deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86\": container with ID starting with deab04d1726b44b3cebb52c39397cf04f08be01a6889a81a01dfaf60368bea86 not found: ID does not exist"
Sep 18 18:46:47.053652 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.053621    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7758489dd-jvpdj"]
Sep 18 18:46:47.060213 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:47.060190    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7758489dd-jvpdj"]
Sep 18 18:46:48.337069 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:46:48.337041    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ddc901-6593-4292-9088-19ab86c9701d" path="/var/lib/kubelet/pods/53ddc901-6593-4292-9088-19ab86c9701d/volumes"
Sep 18 18:47:25.048970 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:25.048905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:25.333476 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:25.333402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:26.083175 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:26.083143    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:47:26.083521 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:26.083415    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="alertmanager" containerID="cri-o://76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5" gracePeriod=120
Sep 18 18:47:26.083589 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:26.083502    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-metric" containerID="cri-o://d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e" gracePeriod=120
Sep 18 18:47:26.083648 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:26.083560    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy" containerID="cri-o://1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b" gracePeriod=120
Sep 18 18:47:26.083648 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:26.083588    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="prom-label-proxy" containerID="cri-o://e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec" gracePeriod=120
Sep 18 18:47:26.083748 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:26.083564    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="config-reloader" containerID="cri-o://a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d" gracePeriod=120
Sep 18 18:47:26.083748 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:26.083512    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-web" containerID="cri-o://4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247" gracePeriod=120
Sep 18 18:47:27.164607 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164576    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerID="e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec" exitCode=0
Sep 18 18:47:27.164994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164611    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerID="1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b" exitCode=0
Sep 18 18:47:27.164994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164623    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerID="a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d" exitCode=0
Sep 18 18:47:27.164994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164631    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerID="76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5" exitCode=0
Sep 18 18:47:27.164994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164631    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"}
Sep 18 18:47:27.164994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"}
Sep 18 18:47:27.164994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"}
Sep 18 18:47:27.164994 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.164761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"}
Sep 18 18:47:27.312502 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.312481    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:27.325641 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325616    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-config-volume\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325730 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-main-db\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325730 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325662    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-web\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325730 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-config-out\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325903 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325733    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-web-config\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325903 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325764    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325903 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325789    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-trusted-ca-bundle\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325903 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325816    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-tls-assets\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325903 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-metrics-client-ca\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.325903 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325894    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-main-tls\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.326194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325918    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf2lg\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-kube-api-access-vf2lg\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.326194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.325972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-cluster-tls-config\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.326194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.326001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy\") pod \"ecce41ba-6178-4be2-955d-2ad715e93edd\" (UID: \"ecce41ba-6178-4be2-955d-2ad715e93edd\") "
Sep 18 18:47:27.326194 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.326049    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:27.326551 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.326198    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:47:27.326551 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.326275    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.326551 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.326295    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-alertmanager-main-db\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.327391 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.327362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:47:27.328646 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.328617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-config-out" (OuterVolumeSpecName: "config-out") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:47:27.328846 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.328803    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:47:27.329100 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.329059    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:47:27.329412 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.329387    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:47:27.329617 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.329575    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-kube-api-access-vf2lg" (OuterVolumeSpecName: "kube-api-access-vf2lg") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "kube-api-access-vf2lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:27.329724 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.329679    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-config-volume" (OuterVolumeSpecName: "config-volume") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:47:27.329912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.329868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:47:27.330417 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.330393    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:47:27.334989 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.334965    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:47:27.339980 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.339960    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-web-config" (OuterVolumeSpecName: "web-config") pod "ecce41ba-6178-4be2-955d-2ad715e93edd" (UID: "ecce41ba-6178-4be2-955d-2ad715e93edd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:47:27.426802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426738    2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-cluster-tls-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.426802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426768    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.426802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426779    2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.426802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426788    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.426802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426796    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ecce41ba-6178-4be2-955d-2ad715e93edd-config-out\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.426802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426805    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-web-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.427055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426813    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.427055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426821    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-tls-assets\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.427055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426829    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecce41ba-6178-4be2-955d-2ad715e93edd-metrics-client-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.427055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426842    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ecce41ba-6178-4be2-955d-2ad715e93edd-secret-alertmanager-main-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:27.427055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:27.426853    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2lg\" (UniqueName: \"kubernetes.io/projected/ecce41ba-6178-4be2-955d-2ad715e93edd-kube-api-access-vf2lg\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:47:28.173103 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.173068    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerID="d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e" exitCode=0
Sep 18 18:47:28.173103 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.173099    2582 generic.go:356] "Generic (PLEG): container finished" podID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerID="4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247" exitCode=0
Sep 18 18:47:28.173432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.173141    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.173432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.173145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"}
Sep 18 18:47:28.173432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.173211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"}
Sep 18 18:47:28.173432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.173238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ecce41ba-6178-4be2-955d-2ad715e93edd","Type":"ContainerDied","Data":"900a4c72c63822e05d5c8c0fabe6864b2596d1f0a75ed3aa54e719d44b5d0e9f"}
Sep 18 18:47:28.173432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.173268    2582 scope.go:117] "RemoveContainer" containerID="e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"
Sep 18 18:47:28.181292 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.181274    2582 scope.go:117] "RemoveContainer" containerID="d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"
Sep 18 18:47:28.187848 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.187830    2582 scope.go:117] "RemoveContainer" containerID="1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"
Sep 18 18:47:28.194048 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.194033    2582 scope.go:117] "RemoveContainer" containerID="4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"
Sep 18 18:47:28.200355 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.200338    2582 scope.go:117] "RemoveContainer" containerID="a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"
Sep 18 18:47:28.201771 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.201753    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:47:28.206840 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.206819    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:47:28.208170 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.208154    2582 scope.go:117] "RemoveContainer" containerID="76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"
Sep 18 18:47:28.214868 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.214851    2582 scope.go:117] "RemoveContainer" containerID="98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c"
Sep 18 18:47:28.221149 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.221135    2582 scope.go:117] "RemoveContainer" containerID="e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"
Sep 18 18:47:28.221398 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:47:28.221380    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec\": container with ID starting with e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec not found: ID does not exist" containerID="e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"
Sep 18 18:47:28.221444 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.221406    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"} err="failed to get container status \"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec\": rpc error: code = NotFound desc = could not find container \"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec\": container with ID starting with e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec not found: ID does not exist"
Sep 18 18:47:28.221444 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.221423    2582 scope.go:117] "RemoveContainer" containerID="d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"
Sep 18 18:47:28.221636 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:47:28.221618    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e\": container with ID starting with d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e not found: ID does not exist" containerID="d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"
Sep 18 18:47:28.221672 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.221643    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"} err="failed to get container status \"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e\": rpc error: code = NotFound desc = could not find container \"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e\": container with ID starting with d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e not found: ID does not exist"
Sep 18 18:47:28.221672 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.221665    2582 scope.go:117] "RemoveContainer" containerID="1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"
Sep 18 18:47:28.221876 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:47:28.221858    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b\": container with ID starting with 1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b not found: ID does not exist" containerID="1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"
Sep 18 18:47:28.221967 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.221906    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"} err="failed to get container status \"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b\": rpc error: code = NotFound desc = could not find container \"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b\": container with ID starting with 1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b not found: ID does not exist"
Sep 18 18:47:28.221967 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.221928    2582 scope.go:117] "RemoveContainer" containerID="4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"
Sep 18 18:47:28.222166 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:47:28.222151    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247\": container with ID starting with 4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247 not found: ID does not exist" containerID="4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"
Sep 18 18:47:28.222199 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222170    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"} err="failed to get container status \"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247\": rpc error: code = NotFound desc = could not find container \"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247\": container with ID starting with 4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247 not found: ID does not exist"
Sep 18 18:47:28.222199 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222183    2582 scope.go:117] "RemoveContainer" containerID="a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"
Sep 18 18:47:28.222376 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:47:28.222360    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d\": container with ID starting with a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d not found: ID does not exist" containerID="a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"
Sep 18 18:47:28.222417 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222381    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"} err="failed to get container status \"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d\": rpc error: code = NotFound desc = could not find container \"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d\": container with ID starting with a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d not found: ID does not exist"
Sep 18 18:47:28.222417 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222393    2582 scope.go:117] "RemoveContainer" containerID="76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"
Sep 18 18:47:28.222586 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:47:28.222570    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5\": container with ID starting with 76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5 not found: ID does not exist" containerID="76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"
Sep 18 18:47:28.222643 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222593    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"} err="failed to get container status \"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5\": rpc error: code = NotFound desc = could not find container \"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5\": container with ID starting with 76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5 not found: ID does not exist"
Sep 18 18:47:28.222643 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222612    2582 scope.go:117] "RemoveContainer" containerID="98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c"
Sep 18 18:47:28.222836 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:47:28.222820    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c\": container with ID starting with 98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c not found: ID does not exist" containerID="98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c"
Sep 18 18:47:28.222872 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222840    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c"} err="failed to get container status \"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c\": rpc error: code = NotFound desc = could not find container \"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c\": container with ID starting with 98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c not found: ID does not exist"
Sep 18 18:47:28.222872 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.222852    2582 scope.go:117] "RemoveContainer" containerID="e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"
Sep 18 18:47:28.223070 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223053    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec"} err="failed to get container status \"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec\": rpc error: code = NotFound desc = could not find container \"e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec\": container with ID starting with e1d8c92f9b62dd35cfb047ce166d38adeac663f531846210fd38a14ddbee7bec not found: ID does not exist"
Sep 18 18:47:28.223116 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223072    2582 scope.go:117] "RemoveContainer" containerID="d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"
Sep 18 18:47:28.223244 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223230    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e"} err="failed to get container status \"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e\": rpc error: code = NotFound desc = could not find container \"d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e\": container with ID starting with d541f4b9d2e92dbe47d523cd23c2fa076a4565a520407cb00b533b1d9e31341e not found: ID does not exist"
Sep 18 18:47:28.223288 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223246    2582 scope.go:117] "RemoveContainer" containerID="1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"
Sep 18 18:47:28.223431 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223413    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b"} err="failed to get container status \"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b\": rpc error: code = NotFound desc = could not find container \"1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b\": container with ID starting with 1f595e59ad3f7e9d099a28a49410847194cf49ef9e748019aa22064bf662f75b not found: ID does not exist"
Sep 18 18:47:28.223496 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223432    2582 scope.go:117] "RemoveContainer" containerID="4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"
Sep 18 18:47:28.223635 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223616    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247"} err="failed to get container status \"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247\": rpc error: code = NotFound desc = could not find container \"4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247\": container with ID starting with 4ae52216548e8da869f43c73448b952fe41d9e6117243e96fb28e8b9f65d5247 not found: ID does not exist"
Sep 18 18:47:28.223684 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223637    2582 scope.go:117] "RemoveContainer" containerID="a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"
Sep 18 18:47:28.223825 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223807    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d"} err="failed to get container status \"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d\": rpc error: code = NotFound desc = could not find container \"a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d\": container with ID starting with a12330612a4942cf1a342ed75243f6afd594d60b5d62315f0b26bddd16871e5d not found: ID does not exist"
Sep 18 18:47:28.223867 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.223826    2582 scope.go:117] "RemoveContainer" containerID="76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"
Sep 18 18:47:28.224019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.224005    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5"} err="failed to get container status \"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5\": rpc error: code = NotFound desc = could not find container \"76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5\": container with ID starting with 76bae9c8961852e19040af6b53cfa5f9f89a9838aeff259b27a7b39b45b733a5 not found: ID does not exist"
Sep 18 18:47:28.224019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.224018    2582 scope.go:117] "RemoveContainer" containerID="98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c"
Sep 18 18:47:28.224214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.224196    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c"} err="failed to get container status \"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c\": rpc error: code = NotFound desc = could not find container \"98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c\": container with ID starting with 98b826b4fba3b432110513a78b4d9e2efb8b9aef44d42f95a098162fa977289c not found: ID does not exist"
Sep 18 18:47:28.240949 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.240924    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:47:28.241160 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241149    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="init-config-reloader"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241161    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="init-config-reloader"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241178    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4889c18-2c2a-4501-b53d-d2fddb6419bb" containerName="console"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241182    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4889c18-2c2a-4501-b53d-d2fddb6419bb" containerName="console"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241190    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ddc901-6593-4292-9088-19ab86c9701d" containerName="console"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241194    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ddc901-6593-4292-9088-19ab86c9701d" containerName="console"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241200    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="alertmanager"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241205    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="alertmanager"
Sep 18 18:47:28.241210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241210    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="config-reloader"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241215    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="config-reloader"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241220    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241225    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241231    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="prom-label-proxy"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241236    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="prom-label-proxy"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241244    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-web"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241249    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-web"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241255    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-metric"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241260    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-metric"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241298    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-metric"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241304    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="config-reloader"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241309    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy-web"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241314    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="kube-rbac-proxy"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241318    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="prom-label-proxy"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241323    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4889c18-2c2a-4501-b53d-d2fddb6419bb" containerName="console"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241329    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ddc901-6593-4292-9088-19ab86c9701d" containerName="console"
Sep 18 18:47:28.241456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.241333    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" containerName="alertmanager"
Sep 18 18:47:28.246165 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.246150    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.248498 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248478    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 18:47:28.248603 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248505    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 18:47:28.248603 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248526    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-66gnc\""
Sep 18 18:47:28.248603 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248578    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 18:47:28.248931 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248911    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 18:47:28.249029 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248970    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 18:47:28.249029 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248978    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 18:47:28.249029 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.248917    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 18:47:28.249158 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.249076    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 18:47:28.254548 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.254530    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 18:47:28.265199 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.265181    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:47:28.332979 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.332949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333077 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.332993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333077 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9957d98-a807-4cec-bd4a-10804ef340d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333077 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktt44\" (UniqueName: \"kubernetes.io/projected/c9957d98-a807-4cec-bd4a-10804ef340d7-kube-api-access-ktt44\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9957d98-a807-4cec-bd4a-10804ef340d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9957d98-a807-4cec-bd4a-10804ef340d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333371 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9957d98-a807-4cec-bd4a-10804ef340d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333371 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333371 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-web-config\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333371 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9957d98-a807-4cec-bd4a-10804ef340d7-config-out\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.333371 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.333361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.336619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.336595    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecce41ba-6178-4be2-955d-2ad715e93edd" path="/var/lib/kubelet/pods/ecce41ba-6178-4be2-955d-2ad715e93edd/volumes"
Sep 18 18:47:28.434373 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434373 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktt44\" (UniqueName: \"kubernetes.io/projected/c9957d98-a807-4cec-bd4a-10804ef340d7-kube-api-access-ktt44\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434373 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9957d98-a807-4cec-bd4a-10804ef340d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434500 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9957d98-a807-4cec-bd4a-10804ef340d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434553 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9957d98-a807-4cec-bd4a-10804ef340d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434673 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434737 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-web-config\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9957d98-a807-4cec-bd4a-10804ef340d7-config-out\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434904 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434904 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.434904 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.434864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9957d98-a807-4cec-bd4a-10804ef340d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.435227 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.435202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9957d98-a807-4cec-bd4a-10804ef340d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.435464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.435364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9957d98-a807-4cec-bd4a-10804ef340d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.435605 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.435478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9957d98-a807-4cec-bd4a-10804ef340d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.437708 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.437558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.437708 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.437612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9957d98-a807-4cec-bd4a-10804ef340d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.437856 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.437718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-web-config\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.437942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.437865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9957d98-a807-4cec-bd4a-10804ef340d7-config-out\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.438058 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.438027    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.438311 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.438206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.438311 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.438292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.438435 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.438373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.439689 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.439664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9957d98-a807-4cec-bd4a-10804ef340d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.443156 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.443136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktt44\" (UniqueName: \"kubernetes.io/projected/c9957d98-a807-4cec-bd4a-10804ef340d7-kube-api-access-ktt44\") pod \"alertmanager-main-0\" (UID: \"c9957d98-a807-4cec-bd4a-10804ef340d7\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.554654 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.554635    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 18:47:28.837138 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:28.837067    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 18:47:28.839864 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:47:28.839840    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9957d98_a807_4cec_bd4a_10804ef340d7.slice/crio-d5b83e2e7cb8c3d9a8e358910eb4c3d3ee284c18dc010a9a93b6d8845c8854d7 WatchSource:0}: Error finding container d5b83e2e7cb8c3d9a8e358910eb4c3d3ee284c18dc010a9a93b6d8845c8854d7: Status 404 returned error can't find the container with id d5b83e2e7cb8c3d9a8e358910eb4c3d3ee284c18dc010a9a93b6d8845c8854d7
Sep 18 18:47:29.177989 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:29.177962    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9957d98-a807-4cec-bd4a-10804ef340d7" containerID="8725a55581a0243ff021ade429e947488aa59f28ac8f34c87cc849b0dae8ccf8" exitCode=0
Sep 18 18:47:29.178306 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:29.178048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerDied","Data":"8725a55581a0243ff021ade429e947488aa59f28ac8f34c87cc849b0dae8ccf8"}
Sep 18 18:47:29.178306 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:29.178118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerStarted","Data":"d5b83e2e7cb8c3d9a8e358910eb4c3d3ee284c18dc010a9a93b6d8845c8854d7"}
Sep 18 18:47:30.148173 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.148147    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"]
Sep 18 18:47:30.151241 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.151226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.153482 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.153464    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 18:47:30.153578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.153493    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 18:47:30.153578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.153494    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 18:47:30.153803 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.153782    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8qqm6\""
Sep 18 18:47:30.153924 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.153802    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 18:47:30.153924 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.153830    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 18:47:30.162962 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.162918    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 18:47:30.171426 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.171407    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"]
Sep 18 18:47:30.186008 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.185983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerStarted","Data":"277268c311c5cd7fbae849da55388fbc7cd43101b74f92b4d0a24272620ffa8a"}
Sep 18 18:47:30.186280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.186034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerStarted","Data":"0fe2dcb7f7702da2a105f6cb9bf0a99ae739d35ba8f6d71e8d7a95b267f51c0a"}
Sep 18 18:47:30.186280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.186058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerStarted","Data":"72685a780c6c575ae411e7deaa302358f87a282b8d4140c64f6e89606a4a5d95"}
Sep 18 18:47:30.186280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.186080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerStarted","Data":"cde29e9f6f30f04cdba9019617ef59da45e22974a2d6ca005fb4fe10a13d28e7"}
Sep 18 18:47:30.186280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.186104    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerStarted","Data":"03dcadef50c1718580234349f0a81916e11926489be8fab4face5090c74bd821"}
Sep 18 18:47:30.186280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.186125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9957d98-a807-4cec-bd4a-10804ef340d7","Type":"ContainerStarted","Data":"f33e2c7ab7ff6e239a98cedacb221ea67f75fac0d74aefa1a4091e0717ab8579"}
Sep 18 18:47:30.246380 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-metrics-client-ca\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.246461 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.246522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-telemeter-client-tls\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.246522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-federate-client-tls\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.246619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.246619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-secret-telemeter-client\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.246731 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-serving-certs-ca-bundle\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.246803 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.246785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2p2\" (UniqueName: \"kubernetes.io/projected/39e6be68-db53-4cc8-b50c-410ee795f389-kube-api-access-gd2p2\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347145 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347248 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-telemeter-client-tls\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347248 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-federate-client-tls\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347437 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347482 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-secret-telemeter-client\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347532 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-serving-certs-ca-bundle\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347580 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2p2\" (UniqueName: \"kubernetes.io/projected/39e6be68-db53-4cc8-b50c-410ee795f389-kube-api-access-gd2p2\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.347634 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.347583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-metrics-client-ca\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.348157 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.348103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.348364 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.348341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-serving-certs-ca-bundle\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.348693 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.348670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39e6be68-db53-4cc8-b50c-410ee795f389-metrics-client-ca\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.349914 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.349851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-telemeter-client-tls\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.350139 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.350123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-secret-telemeter-client\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.350276 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.350254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.350691 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.350662    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/39e6be68-db53-4cc8-b50c-410ee795f389-federate-client-tls\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.358230 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.358214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2p2\" (UniqueName: \"kubernetes.io/projected/39e6be68-db53-4cc8-b50c-410ee795f389-kube-api-access-gd2p2\") pod \"telemeter-client-6b8db5cb94-xx68t\" (UID: \"39e6be68-db53-4cc8-b50c-410ee795f389\") " pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.469089 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.469032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"
Sep 18 18:47:30.754576 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.754499    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.738885149 podStartE2EDuration="2.754481089s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.59594ms" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:30.221007616 +0000 UTC m=+174.013684565" watchObservedRunningTime="2026-09-18 18:47:30.754481089 +0000 UTC m=+174.547158019"
Sep 18 18:47:30.754839 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:30.754826    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b8db5cb94-xx68t"]
Sep 18 18:47:30.757451 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:47:30.757426    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e6be68_db53_4cc8_b50c_410ee795f389.slice/crio-b059c79bb86172444ef2e1325ea1d582c93cce19b881fdfa90bd488127e91f85 WatchSource:0}: Error finding container b059c79bb86172444ef2e1325ea1d582c93cce19b881fdfa90bd488127e91f85: Status 404 returned error can't find the container with id b059c79bb86172444ef2e1325ea1d582c93cce19b881fdfa90bd488127e91f85
Sep 18 18:47:31.191711 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:31.191682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t" event={"ID":"39e6be68-db53-4cc8-b50c-410ee795f389","Type":"ContainerStarted","Data":"b059c79bb86172444ef2e1325ea1d582c93cce19b881fdfa90bd488127e91f85"}
Sep 18 18:47:33.206804 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.206770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t" event={"ID":"39e6be68-db53-4cc8-b50c-410ee795f389","Type":"ContainerStarted","Data":"1d895bff1d3382404f4d3510a0420bd067e812842803002d864aad7a85ebdacd"}
Sep 18 18:47:33.207133 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.206850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t" event={"ID":"39e6be68-db53-4cc8-b50c-410ee795f389","Type":"ContainerStarted","Data":"5628daa4e182cde96761526e43fc65b36ca5df0c86118b93d55d4833f718a945"}
Sep 18 18:47:33.207133 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.206877    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t" event={"ID":"39e6be68-db53-4cc8-b50c-410ee795f389","Type":"ContainerStarted","Data":"fa832d101e1133d1e53dcb92586ebf1cf9cc3a797c92ea9ec9f662ce63f37063"}
Sep 18 18:47:33.232575 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.232533    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b8db5cb94-xx68t" podStartSLOduration=1.621852744 podStartE2EDuration="3.23251947s" totalImagesPullingTime="1.610666726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:33.230413049 +0000 UTC m=+177.023089979" watchObservedRunningTime="2026-09-18 18:47:33.23251947 +0000 UTC m=+177.025196399"
Sep 18 18:47:33.316756 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.316727    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-747765b5-82swm"]
Sep 18 18:47:33.319976 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.319960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.335008 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.334990    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747765b5-82swm"]
Sep 18 18:47:33.368357 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.368336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-trusted-ca-bundle\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.368463 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.368367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-oauth-serving-cert\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.368463 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.368390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-serving-cert\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.368463 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.368424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jp7\" (UniqueName: \"kubernetes.io/projected/44340262-b460-49f4-afaf-075527a5d246-kube-api-access-s7jp7\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.368463 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.368459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-console-config\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.368624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.368474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-oauth-config\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.368624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.368488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-service-ca\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.381847 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.381817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:33.456392 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.456368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:33.469439 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.469394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-console-config\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.469439 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.469421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-oauth-config\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.469439 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.469435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-service-ca\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.469610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.469453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-trusted-ca-bundle\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.469610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.469589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-oauth-serving-cert\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.469710 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.469636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-serving-cert\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.469710 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.469692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jp7\" (UniqueName: \"kubernetes.io/projected/44340262-b460-49f4-afaf-075527a5d246-kube-api-access-s7jp7\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.470263 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.470176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-service-ca\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.470263 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.470226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-console-config\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.470435 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.470337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-trusted-ca-bundle\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.470435 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.470338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-oauth-serving-cert\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.471984 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.471967    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-oauth-config\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.472059 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.472042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-serving-cert\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.478644 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.478627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jp7\" (UniqueName: \"kubernetes.io/projected/44340262-b460-49f4-afaf-075527a5d246-kube-api-access-s7jp7\") pod \"console-747765b5-82swm\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") " pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.529254 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.529231    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:33.597043 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.597018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:33.627964 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.627945    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:33.898479 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:33.898450    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-747765b5-82swm"]
Sep 18 18:47:33.901026 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:47:33.901001    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44340262_b460_49f4_afaf_075527a5d246.slice/crio-fbc1b8821357979381d959c66f2a92708429fa477c5ad8e8c749e2860aef46c8 WatchSource:0}: Error finding container fbc1b8821357979381d959c66f2a92708429fa477c5ad8e8c749e2860aef46c8: Status 404 returned error can't find the container with id fbc1b8821357979381d959c66f2a92708429fa477c5ad8e8c749e2860aef46c8
Sep 18 18:47:34.211597 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:34.211513    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747765b5-82swm" event={"ID":"44340262-b460-49f4-afaf-075527a5d246","Type":"ContainerStarted","Data":"7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7"}
Sep 18 18:47:34.211597 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:34.211576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747765b5-82swm" event={"ID":"44340262-b460-49f4-afaf-075527a5d246","Type":"ContainerStarted","Data":"fbc1b8821357979381d959c66f2a92708429fa477c5ad8e8c749e2860aef46c8"}
Sep 18 18:47:34.230062 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:34.229405    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-747765b5-82swm" podStartSLOduration=1.229387958 podStartE2EDuration="1.229387958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:47:34.229084065 +0000 UTC m=+178.021760994" watchObservedRunningTime="2026-09-18 18:47:34.229387958 +0000 UTC m=+178.022064888"
Sep 18 18:47:34.449819 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:34.449795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:34.526653 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:34.526595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:43.628845 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:43.628811    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:43.629201 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:43.628864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:43.633025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:43.633002    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:44.249076 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:44.249053    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:47:44.300354 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:44.300331    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c9dbf984-nnk5d"]
Sep 18 18:47:44.412179 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:44.412155    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:44.512832 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:44.512777    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:47:44.613704 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:47:44.613684    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:48:09.318641 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.318600    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59c9dbf984-nnk5d" podUID="d4f722a7-b8c9-4576-818d-eba1840cea42" containerName="console" containerID="cri-o://353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac" gracePeriod=15
Sep 18 18:48:09.547284 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.547258    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:48:09.615403 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615345    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcdjm\" (UniqueName: \"kubernetes.io/projected/d4f722a7-b8c9-4576-818d-eba1840cea42-kube-api-access-tcdjm\") pod \"d4f722a7-b8c9-4576-818d-eba1840cea42\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") "
Sep 18 18:48:09.615403 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615383    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-service-ca\") pod \"d4f722a7-b8c9-4576-818d-eba1840cea42\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") "
Sep 18 18:48:09.615570 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-oauth-serving-cert\") pod \"d4f722a7-b8c9-4576-818d-eba1840cea42\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") "
Sep 18 18:48:09.615570 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-trusted-ca-bundle\") pod \"d4f722a7-b8c9-4576-818d-eba1840cea42\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") "
Sep 18 18:48:09.615671 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615573    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-serving-cert\") pod \"d4f722a7-b8c9-4576-818d-eba1840cea42\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") "
Sep 18 18:48:09.615671 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-oauth-config\") pod \"d4f722a7-b8c9-4576-818d-eba1840cea42\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") "
Sep 18 18:48:09.615761 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-console-config\") pod \"d4f722a7-b8c9-4576-818d-eba1840cea42\" (UID: \"d4f722a7-b8c9-4576-818d-eba1840cea42\") "
Sep 18 18:48:09.615825 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615797    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-service-ca" (OuterVolumeSpecName: "service-ca") pod "d4f722a7-b8c9-4576-818d-eba1840cea42" (UID: "d4f722a7-b8c9-4576-818d-eba1840cea42"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:48:09.615825 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615811    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d4f722a7-b8c9-4576-818d-eba1840cea42" (UID: "d4f722a7-b8c9-4576-818d-eba1840cea42"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:48:09.615954 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615916    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d4f722a7-b8c9-4576-818d-eba1840cea42" (UID: "d4f722a7-b8c9-4576-818d-eba1840cea42"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:48:09.616008 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615953    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:48:09.616008 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.615967    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-oauth-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:48:09.616174 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.616154    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-console-config" (OuterVolumeSpecName: "console-config") pod "d4f722a7-b8c9-4576-818d-eba1840cea42" (UID: "d4f722a7-b8c9-4576-818d-eba1840cea42"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:48:09.617514 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.617487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f722a7-b8c9-4576-818d-eba1840cea42-kube-api-access-tcdjm" (OuterVolumeSpecName: "kube-api-access-tcdjm") pod "d4f722a7-b8c9-4576-818d-eba1840cea42" (UID: "d4f722a7-b8c9-4576-818d-eba1840cea42"). InnerVolumeSpecName "kube-api-access-tcdjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:48:09.617661 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.617636    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d4f722a7-b8c9-4576-818d-eba1840cea42" (UID: "d4f722a7-b8c9-4576-818d-eba1840cea42"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:48:09.617834 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.617814    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d4f722a7-b8c9-4576-818d-eba1840cea42" (UID: "d4f722a7-b8c9-4576-818d-eba1840cea42"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:48:09.716332 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.716300    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-oauth-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:48:09.716332 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.716330    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-console-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:48:09.716332 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.716340    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdjm\" (UniqueName: \"kubernetes.io/projected/d4f722a7-b8c9-4576-818d-eba1840cea42-kube-api-access-tcdjm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:48:09.716476 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.716350    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f722a7-b8c9-4576-818d-eba1840cea42-trusted-ca-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:48:09.716476 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:09.716358    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4f722a7-b8c9-4576-818d-eba1840cea42-console-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:48:10.329631 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.329606    2582 generic.go:356] "Generic (PLEG): container finished" podID="d4f722a7-b8c9-4576-818d-eba1840cea42" containerID="353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac" exitCode=0
Sep 18 18:48:10.329946 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.329710    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9dbf984-nnk5d"
Sep 18 18:48:10.335445 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.335422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9dbf984-nnk5d" event={"ID":"d4f722a7-b8c9-4576-818d-eba1840cea42","Type":"ContainerDied","Data":"353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac"}
Sep 18 18:48:10.335534 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.335460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9dbf984-nnk5d" event={"ID":"d4f722a7-b8c9-4576-818d-eba1840cea42","Type":"ContainerDied","Data":"3a9e0179226034c00901b16962de3994b044a6b2b5545d2e63093c70e3ee8749"}
Sep 18 18:48:10.335534 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.335479    2582 scope.go:117] "RemoveContainer" containerID="353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac"
Sep 18 18:48:10.344149 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.344130    2582 scope.go:117] "RemoveContainer" containerID="353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac"
Sep 18 18:48:10.344399 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:48:10.344381    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac\": container with ID starting with 353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac not found: ID does not exist" containerID="353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac"
Sep 18 18:48:10.344444 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.344406    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac"} err="failed to get container status \"353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac\": rpc error: code = NotFound desc = could not find container \"353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac\": container with ID starting with 353f29e9d20ff6f80292b621f75e5631e4137f45f7b43aef35f592ec432c3aac not found: ID does not exist"
Sep 18 18:48:10.360617 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.360458    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c9dbf984-nnk5d"]
Sep 18 18:48:10.364843 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:10.364825    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59c9dbf984-nnk5d"]
Sep 18 18:48:12.333161 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:12.333134    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f722a7-b8c9-4576-818d-eba1840cea42" path="/var/lib/kubelet/pods/d4f722a7-b8c9-4576-818d-eba1840cea42/volumes"
Sep 18 18:48:49.990331 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:49.990251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:48:50.360663 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:50.360592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:48:50.442231 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:50.442206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:48:54.479380 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.479348    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85bf49897c-7lzx4"]
Sep 18 18:48:54.479723 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.479606    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f722a7-b8c9-4576-818d-eba1840cea42" containerName="console"
Sep 18 18:48:54.479723 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.479616    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f722a7-b8c9-4576-818d-eba1840cea42" containerName="console"
Sep 18 18:48:54.479723 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.479677    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f722a7-b8c9-4576-818d-eba1840cea42" containerName="console"
Sep 18 18:48:54.482465 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.482436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.513903 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.513858    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85bf49897c-7lzx4"]
Sep 18 18:48:54.575642 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.575620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:48:54.607445 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.607421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-oauth-serving-cert\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.607523 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.607452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-oauth-config\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.607523 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.607487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-serving-cert\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.607589 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.607533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-config\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.607589 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.607548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-service-ca\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.607589 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.607561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxrq\" (UniqueName: \"kubernetes.io/projected/513ee270-d8fd-42a4-aae0-f47a2be252fc-kube-api-access-fnxrq\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.607684 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.607611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-trusted-ca-bundle\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.677334 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.677308    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:48:54.708486 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.708466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-oauth-config\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.708545 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.708514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-serving-cert\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.708545 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.708540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-config\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.708681 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.708664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-service-ca\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.708720 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.708697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxrq\" (UniqueName: \"kubernetes.io/projected/513ee270-d8fd-42a4-aae0-f47a2be252fc-kube-api-access-fnxrq\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.708771 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.708727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-trusted-ca-bundle\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.708771 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.708761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-oauth-serving-cert\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.709271 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.709245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-config\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.709382 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.709358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-oauth-serving-cert\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.709570 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.709544    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-service-ca\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.709616 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.709545    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-trusted-ca-bundle\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.710958 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.710936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-oauth-config\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.711175 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.711156    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-serving-cert\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.718863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.718832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxrq\" (UniqueName: \"kubernetes.io/projected/513ee270-d8fd-42a4-aae0-f47a2be252fc-kube-api-access-fnxrq\") pod \"console-85bf49897c-7lzx4\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") " pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:54.791255 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:54.791207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:48:55.060121 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:55.060054    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85bf49897c-7lzx4"]
Sep 18 18:48:55.062865 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:48:55.062838    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513ee270_d8fd_42a4_aae0_f47a2be252fc.slice/crio-3599858599b4decefd37f34c0f8c816ee9a363cf7bfbd113cef2fb54971151e0 WatchSource:0}: Error finding container 3599858599b4decefd37f34c0f8c816ee9a363cf7bfbd113cef2fb54971151e0: Status 404 returned error can't find the container with id 3599858599b4decefd37f34c0f8c816ee9a363cf7bfbd113cef2fb54971151e0
Sep 18 18:48:55.475164 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:55.475128    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bf49897c-7lzx4" event={"ID":"513ee270-d8fd-42a4-aae0-f47a2be252fc","Type":"ContainerStarted","Data":"744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f"}
Sep 18 18:48:55.475355 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:55.475185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bf49897c-7lzx4" event={"ID":"513ee270-d8fd-42a4-aae0-f47a2be252fc","Type":"ContainerStarted","Data":"3599858599b4decefd37f34c0f8c816ee9a363cf7bfbd113cef2fb54971151e0"}
Sep 18 18:48:55.493651 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:48:55.493603    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85bf49897c-7lzx4" podStartSLOduration=1.493588679 podStartE2EDuration="1.493588679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:48:55.493173653 +0000 UTC m=+259.285850581" watchObservedRunningTime="2026-09-18 18:48:55.493588679 +0000 UTC m=+259.286265620"
Sep 18 18:49:02.385039 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:02.385009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:02.462727 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:02.462694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:04.791809 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:04.791780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:49:04.792254 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:04.791830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:49:04.795791 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:04.795768    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:49:05.511187 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:05.511164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:49:05.559518 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:05.559490    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747765b5-82swm"]
Sep 18 18:49:05.673754 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:05.673729    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:05.773721 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:05.773658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:05.867355 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:05.867326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:16.255522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:16.255416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:16.347780 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:16.347720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:16.623659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:16.623605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:30.578415 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.578374    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-747765b5-82swm" podUID="44340262-b460-49f4-afaf-075527a5d246" containerName="console" containerID="cri-o://7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7" gracePeriod=15
Sep 18 18:49:30.818323 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.818300    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:49:30.941805 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.941779    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-trusted-ca-bundle\") pod \"44340262-b460-49f4-afaf-075527a5d246\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") "
Sep 18 18:49:30.941931 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.941825    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-oauth-config\") pod \"44340262-b460-49f4-afaf-075527a5d246\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") "
Sep 18 18:49:30.941931 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.941845    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7jp7\" (UniqueName: \"kubernetes.io/projected/44340262-b460-49f4-afaf-075527a5d246-kube-api-access-s7jp7\") pod \"44340262-b460-49f4-afaf-075527a5d246\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") "
Sep 18 18:49:30.941931 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.941892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-oauth-serving-cert\") pod \"44340262-b460-49f4-afaf-075527a5d246\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") "
Sep 18 18:49:30.941931 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.941913    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-serving-cert\") pod \"44340262-b460-49f4-afaf-075527a5d246\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") "
Sep 18 18:49:30.942142 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.941932    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-service-ca\") pod \"44340262-b460-49f4-afaf-075527a5d246\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") "
Sep 18 18:49:30.942142 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.941967    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-console-config\") pod \"44340262-b460-49f4-afaf-075527a5d246\" (UID: \"44340262-b460-49f4-afaf-075527a5d246\") "
Sep 18 18:49:30.942427 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.942334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "44340262-b460-49f4-afaf-075527a5d246" (UID: "44340262-b460-49f4-afaf-075527a5d246"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:49:30.942427 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.942386    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-service-ca" (OuterVolumeSpecName: "service-ca") pod "44340262-b460-49f4-afaf-075527a5d246" (UID: "44340262-b460-49f4-afaf-075527a5d246"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:49:30.942745 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.942451    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-console-config" (OuterVolumeSpecName: "console-config") pod "44340262-b460-49f4-afaf-075527a5d246" (UID: "44340262-b460-49f4-afaf-075527a5d246"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:49:30.942745 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.942472    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "44340262-b460-49f4-afaf-075527a5d246" (UID: "44340262-b460-49f4-afaf-075527a5d246"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:49:30.944100 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.944077    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "44340262-b460-49f4-afaf-075527a5d246" (UID: "44340262-b460-49f4-afaf-075527a5d246"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:49:30.944200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.944133    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44340262-b460-49f4-afaf-075527a5d246-kube-api-access-s7jp7" (OuterVolumeSpecName: "kube-api-access-s7jp7") pod "44340262-b460-49f4-afaf-075527a5d246" (UID: "44340262-b460-49f4-afaf-075527a5d246"). InnerVolumeSpecName "kube-api-access-s7jp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:49:30.944200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:30.944176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "44340262-b460-49f4-afaf-075527a5d246" (UID: "44340262-b460-49f4-afaf-075527a5d246"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:49:31.043244 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.043223    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-trusted-ca-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:31.043244 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.043242    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-oauth-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:31.043393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.043251    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jp7\" (UniqueName: \"kubernetes.io/projected/44340262-b460-49f4-afaf-075527a5d246-kube-api-access-s7jp7\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:31.043393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.043260    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-oauth-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:31.043393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.043268    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44340262-b460-49f4-afaf-075527a5d246-console-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:31.043393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.043276    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:31.043393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.043283    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44340262-b460-49f4-afaf-075527a5d246-console-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:31.593427 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.593402    2582 generic.go:356] "Generic (PLEG): container finished" podID="44340262-b460-49f4-afaf-075527a5d246" containerID="7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7" exitCode=0
Sep 18 18:49:31.593703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.593441    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-747765b5-82swm"
Sep 18 18:49:31.593703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.593454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747765b5-82swm" event={"ID":"44340262-b460-49f4-afaf-075527a5d246","Type":"ContainerDied","Data":"7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7"}
Sep 18 18:49:31.593703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.593489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-747765b5-82swm" event={"ID":"44340262-b460-49f4-afaf-075527a5d246","Type":"ContainerDied","Data":"fbc1b8821357979381d959c66f2a92708429fa477c5ad8e8c749e2860aef46c8"}
Sep 18 18:49:31.593703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.593507    2582 scope.go:117] "RemoveContainer" containerID="7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7"
Sep 18 18:49:31.601580 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.601565    2582 scope.go:117] "RemoveContainer" containerID="7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7"
Sep 18 18:49:31.601971 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:49:31.601860    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7\": container with ID starting with 7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7 not found: ID does not exist" containerID="7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7"
Sep 18 18:49:31.601971 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.601922    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7"} err="failed to get container status \"7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7\": rpc error: code = NotFound desc = could not find container \"7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7\": container with ID starting with 7a855831de916252f20bdcf585405ccd6c19f3bec09b7363a8c60f0bb3b3b4f7 not found: ID does not exist"
Sep 18 18:49:31.613325 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.613306    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-747765b5-82swm"]
Sep 18 18:49:31.617653 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:31.617634    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-747765b5-82swm"]
Sep 18 18:49:32.333145 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:32.333125    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44340262-b460-49f4-afaf-075527a5d246" path="/var/lib/kubelet/pods/44340262-b460-49f4-afaf-075527a5d246/volumes"
Sep 18 18:49:38.036084 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:38.036056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:38.140584 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:38.140558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:49:38.230130 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:38.230109    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 18:49:39.441673 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.441644    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"]
Sep 18 18:49:39.443948 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.441912    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44340262-b460-49f4-afaf-075527a5d246" containerName="console"
Sep 18 18:49:39.443948 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.441922    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44340262-b460-49f4-afaf-075527a5d246" containerName="console"
Sep 18 18:49:39.443948 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.441966    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44340262-b460-49f4-afaf-075527a5d246" containerName="console"
Sep 18 18:49:39.444836 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.444822    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.449237 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.449216    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:49:39.449330 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.449220    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:49:39.449982 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.449966    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mxnzd\""
Sep 18 18:49:39.463320 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.463292    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"]
Sep 18 18:49:39.601919 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.601896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24gd\" (UniqueName: \"kubernetes.io/projected/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-kube-api-access-c24gd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.602008 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.601924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.602008 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.601953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.702537 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.702488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24gd\" (UniqueName: \"kubernetes.io/projected/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-kube-api-access-c24gd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.702537 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.702513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.702537 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.702532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.702927 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.702908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.702927 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.702922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.712374 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.712353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24gd\" (UniqueName: \"kubernetes.io/projected/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-kube-api-access-c24gd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:39.753689 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:39.753669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:40.022531 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:40.022508    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"]
Sep 18 18:49:40.024462 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:49:40.024435    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242e7b2f_0af0_4d6a_81ae_69e7cf901a29.slice/crio-262aae0b912e7bab047221a513a8368acb34dfad8a15439fb7fd06586db7b708 WatchSource:0}: Error finding container 262aae0b912e7bab047221a513a8368acb34dfad8a15439fb7fd06586db7b708: Status 404 returned error can't find the container with id 262aae0b912e7bab047221a513a8368acb34dfad8a15439fb7fd06586db7b708
Sep 18 18:49:40.026536 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:40.026519    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:49:40.627497 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:40.627469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt" event={"ID":"242e7b2f-0af0-4d6a-81ae-69e7cf901a29","Type":"ContainerStarted","Data":"262aae0b912e7bab047221a513a8368acb34dfad8a15439fb7fd06586db7b708"}
Sep 18 18:49:46.653055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:46.653018    2582 generic.go:356] "Generic (PLEG): container finished" podID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerID="37629d26d1d619f77be6ad30c44a72465b11f75a544b859de74e5107df62b8fe" exitCode=0
Sep 18 18:49:46.653340 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:46.653091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt" event={"ID":"242e7b2f-0af0-4d6a-81ae-69e7cf901a29","Type":"ContainerDied","Data":"37629d26d1d619f77be6ad30c44a72465b11f75a544b859de74e5107df62b8fe"}
Sep 18 18:49:49.667089 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:49.667057    2582 generic.go:356] "Generic (PLEG): container finished" podID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerID="d80ddd079c046ffa53d13925a3da6322efa8aeb2f79a7d7c2bee92db17c4ed73" exitCode=0
Sep 18 18:49:49.667489 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:49.667098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt" event={"ID":"242e7b2f-0af0-4d6a-81ae-69e7cf901a29","Type":"ContainerDied","Data":"d80ddd079c046ffa53d13925a3da6322efa8aeb2f79a7d7c2bee92db17c4ed73"}
Sep 18 18:49:57.702866 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:57.702835    2582 generic.go:356] "Generic (PLEG): container finished" podID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerID="f567ece3fea7773d300f8c0f57ac3a083c292fd4947d19674e8eb1b7ebc0afeb" exitCode=0
Sep 18 18:49:57.703157 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:57.702914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt" event={"ID":"242e7b2f-0af0-4d6a-81ae-69e7cf901a29","Type":"ContainerDied","Data":"f567ece3fea7773d300f8c0f57ac3a083c292fd4947d19674e8eb1b7ebc0afeb"}
Sep 18 18:49:58.821126 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.821102    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:58.835118 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.835096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-util\") pod \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") "
Sep 18 18:49:58.835237 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.835139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c24gd\" (UniqueName: \"kubernetes.io/projected/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-kube-api-access-c24gd\") pod \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") "
Sep 18 18:49:58.835237 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.835207    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-bundle\") pod \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\" (UID: \"242e7b2f-0af0-4d6a-81ae-69e7cf901a29\") "
Sep 18 18:49:58.835932 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.835903    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-bundle" (OuterVolumeSpecName: "bundle") pod "242e7b2f-0af0-4d6a-81ae-69e7cf901a29" (UID: "242e7b2f-0af0-4d6a-81ae-69e7cf901a29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:49:58.837369 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.837341    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-kube-api-access-c24gd" (OuterVolumeSpecName: "kube-api-access-c24gd") pod "242e7b2f-0af0-4d6a-81ae-69e7cf901a29" (UID: "242e7b2f-0af0-4d6a-81ae-69e7cf901a29"). InnerVolumeSpecName "kube-api-access-c24gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:49:58.839421 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.839401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-util" (OuterVolumeSpecName: "util") pod "242e7b2f-0af0-4d6a-81ae-69e7cf901a29" (UID: "242e7b2f-0af0-4d6a-81ae-69e7cf901a29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:49:58.935653 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.935631    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-util\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:58.935653 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.935653    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24gd\" (UniqueName: \"kubernetes.io/projected/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-kube-api-access-c24gd\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:58.935772 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:58.935663    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/242e7b2f-0af0-4d6a-81ae-69e7cf901a29-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:49:59.711337 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:59.711309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt" event={"ID":"242e7b2f-0af0-4d6a-81ae-69e7cf901a29","Type":"ContainerDied","Data":"262aae0b912e7bab047221a513a8368acb34dfad8a15439fb7fd06586db7b708"}
Sep 18 18:49:59.711337 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:59.711327    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2wlt"
Sep 18 18:49:59.711477 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:49:59.711346    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262aae0b912e7bab047221a513a8368acb34dfad8a15439fb7fd06586db7b708"
Sep 18 18:50:05.599840 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:05.599781    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:07.439348 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:07.439300    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:09.559719 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:09.559681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:09.656364 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:09.656338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:11.705229 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:11.705197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:11.839552 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:11.839524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:12.113499 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113438    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5lhn8"]
Sep 18 18:50:12.113748 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113737    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerName="extract"
Sep 18 18:50:12.113787 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113750    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerName="extract"
Sep 18 18:50:12.113787 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113764    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerName="util"
Sep 18 18:50:12.113787 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113769    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerName="util"
Sep 18 18:50:12.113787 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113779    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerName="pull"
Sep 18 18:50:12.113787 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113785    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerName="pull"
Sep 18 18:50:12.113950 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.113837    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="242e7b2f-0af0-4d6a-81ae-69e7cf901a29" containerName="extract"
Sep 18 18:50:12.117107 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.117086    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.119861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.119839    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hg5wm\""
Sep 18 18:50:12.120450 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.120432    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 18:50:12.120517 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.120435    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 18:50:12.134464 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.134445    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5lhn8"]
Sep 18 18:50:12.172053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.172016    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:12.222678 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.222656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxrv\" (UniqueName: \"kubernetes.io/projected/0face2a9-083f-4fc8-b653-7b70f2833e77-kube-api-access-jjxrv\") pod \"cert-manager-webhook-6486ffd997-5lhn8\" (UID: \"0face2a9-083f-4fc8-b653-7b70f2833e77\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.222758 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.222689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0face2a9-083f-4fc8-b653-7b70f2833e77-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5lhn8\" (UID: \"0face2a9-083f-4fc8-b653-7b70f2833e77\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.304160 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.304142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:12.323353 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.323333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxrv\" (UniqueName: \"kubernetes.io/projected/0face2a9-083f-4fc8-b653-7b70f2833e77-kube-api-access-jjxrv\") pod \"cert-manager-webhook-6486ffd997-5lhn8\" (UID: \"0face2a9-083f-4fc8-b653-7b70f2833e77\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.323419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.323363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0face2a9-083f-4fc8-b653-7b70f2833e77-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5lhn8\" (UID: \"0face2a9-083f-4fc8-b653-7b70f2833e77\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.333037 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.333019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxrv\" (UniqueName: \"kubernetes.io/projected/0face2a9-083f-4fc8-b653-7b70f2833e77-kube-api-access-jjxrv\") pod \"cert-manager-webhook-6486ffd997-5lhn8\" (UID: \"0face2a9-083f-4fc8-b653-7b70f2833e77\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.336063 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.336044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0face2a9-083f-4fc8-b653-7b70f2833e77-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5lhn8\" (UID: \"0face2a9-083f-4fc8-b653-7b70f2833e77\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.425210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.425192    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:12.733210 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.733151    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5lhn8"]
Sep 18 18:50:12.735937 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:50:12.735907    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0face2a9_083f_4fc8_b653_7b70f2833e77.slice/crio-49d852925bf4752d92a4a72f5dd4b99f32092644106be9d474c7e01cecdccb6c WatchSource:0}: Error finding container 49d852925bf4752d92a4a72f5dd4b99f32092644106be9d474c7e01cecdccb6c: Status 404 returned error can't find the container with id 49d852925bf4752d92a4a72f5dd4b99f32092644106be9d474c7e01cecdccb6c
Sep 18 18:50:12.755149 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:12.755125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8" event={"ID":"0face2a9-083f-4fc8-b653-7b70f2833e77","Type":"ContainerStarted","Data":"49d852925bf4752d92a4a72f5dd4b99f32092644106be9d474c7e01cecdccb6c"}
Sep 18 18:50:14.098572 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.098538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:14.309213 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.309180    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:14.417914 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.417865    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"]
Sep 18 18:50:14.420121 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.420034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:14.422314 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.422291    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-prk4z\""
Sep 18 18:50:14.435694 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.435673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"]
Sep 18 18:50:14.439192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.439172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b87cd79-1913-4cf0-8d61-05f527b6aafa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qpg9s\" (UID: \"1b87cd79-1913-4cf0-8d61-05f527b6aafa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:14.439304 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.439233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xn6\" (UniqueName: \"kubernetes.io/projected/1b87cd79-1913-4cf0-8d61-05f527b6aafa-kube-api-access-j5xn6\") pod \"cert-manager-cainjector-559cdd8c87-qpg9s\" (UID: \"1b87cd79-1913-4cf0-8d61-05f527b6aafa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:14.540161 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.540135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b87cd79-1913-4cf0-8d61-05f527b6aafa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qpg9s\" (UID: \"1b87cd79-1913-4cf0-8d61-05f527b6aafa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:14.540305 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.540198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xn6\" (UniqueName: \"kubernetes.io/projected/1b87cd79-1913-4cf0-8d61-05f527b6aafa-kube-api-access-j5xn6\") pod \"cert-manager-cainjector-559cdd8c87-qpg9s\" (UID: \"1b87cd79-1913-4cf0-8d61-05f527b6aafa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:14.549531 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.549503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xn6\" (UniqueName: \"kubernetes.io/projected/1b87cd79-1913-4cf0-8d61-05f527b6aafa-kube-api-access-j5xn6\") pod \"cert-manager-cainjector-559cdd8c87-qpg9s\" (UID: \"1b87cd79-1913-4cf0-8d61-05f527b6aafa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:14.549649 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.549576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b87cd79-1913-4cf0-8d61-05f527b6aafa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qpg9s\" (UID: \"1b87cd79-1913-4cf0-8d61-05f527b6aafa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:14.729701 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:14.729634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"
Sep 18 18:50:15.545181 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:15.544952    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s"]
Sep 18 18:50:15.547440 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:50:15.547407    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b87cd79_1913_4cf0_8d61_05f527b6aafa.slice/crio-0a523d639f664876288978cb39b0a708264410d9d5d35b148dffc9dbf8c5d844 WatchSource:0}: Error finding container 0a523d639f664876288978cb39b0a708264410d9d5d35b148dffc9dbf8c5d844: Status 404 returned error can't find the container with id 0a523d639f664876288978cb39b0a708264410d9d5d35b148dffc9dbf8c5d844
Sep 18 18:50:15.769040 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:15.769001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s" event={"ID":"1b87cd79-1913-4cf0-8d61-05f527b6aafa","Type":"ContainerStarted","Data":"ff429a1543334db5cb43625c95830e07e5fde1d5b433afa53e5045b45683248c"}
Sep 18 18:50:15.769215 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:15.769053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s" event={"ID":"1b87cd79-1913-4cf0-8d61-05f527b6aafa","Type":"ContainerStarted","Data":"0a523d639f664876288978cb39b0a708264410d9d5d35b148dffc9dbf8c5d844"}
Sep 18 18:50:15.770693 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:15.770667    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8" event={"ID":"0face2a9-083f-4fc8-b653-7b70f2833e77","Type":"ContainerStarted","Data":"e523a0c717f57c061177344e6da141c5b5de7b0afd29b604cefd597bc6c962ec"}
Sep 18 18:50:15.770816 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:15.770778    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:15.788648 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:15.788582    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qpg9s" podStartSLOduration=1.788564357 podStartE2EDuration="1.788564357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:50:15.786080315 +0000 UTC m=+339.578757262" watchObservedRunningTime="2026-09-18 18:50:15.788564357 +0000 UTC m=+339.581241289"
Sep 18 18:50:15.803709 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:15.803630    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8" podStartSLOduration=1.20829744 podStartE2EDuration="3.803616263s" totalImagesPullingTime="2.595318823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:50:15.802470544 +0000 UTC m=+339.595147474" watchObservedRunningTime="2026-09-18 18:50:15.803616263 +0000 UTC m=+339.596293191"
Sep 18 18:50:16.364857 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:16.364827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:16.461787 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:16.461761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:21.286394 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:21.286319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:21.776483 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:21.776455    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5lhn8"
Sep 18 18:50:32.837787 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:32.837755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:32.935044 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:32.935016    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:33.522816 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:33.522790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:33.619461 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:33.619444    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:39.471011 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:39.470968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:39.550255 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:39.550226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:42.701405 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:42.701373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:42.804401 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:42.804372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:44.135061 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.135030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:44.299860 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.299828    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"]
Sep 18 18:50:44.302278 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.302259    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.304680 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.304651    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:50:44.305556 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.305537    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:50:44.305644 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.305559    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mxnzd\""
Sep 18 18:50:44.314651 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.314624    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"]
Sep 18 18:50:44.347801 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.347780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.347934 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.347816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qb5\" (UniqueName: \"kubernetes.io/projected/b9d84edd-42f7-403e-a1fb-743b4e91b329-kube-api-access-t2qb5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.348003 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.347938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.449049 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.449024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.449153 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.449059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.449153 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.449083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qb5\" (UniqueName: \"kubernetes.io/projected/b9d84edd-42f7-403e-a1fb-743b4e91b329-kube-api-access-t2qb5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.449431 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.449412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.449494 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.449435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.462367 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.462342    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qb5\" (UniqueName: \"kubernetes.io/projected/b9d84edd-42f7-403e-a1fb-743b4e91b329-kube-api-access-t2qb5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.611348 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.611327    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:44.880381 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:44.880315    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"]
Sep 18 18:50:44.883457 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:50:44.883432    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d84edd_42f7_403e_a1fb_743b4e91b329.slice/crio-724ce1ed1dc1d1cf91e1fabca7de63c1cadea0a7c4937c353646cd4f32d67e61 WatchSource:0}: Error finding container 724ce1ed1dc1d1cf91e1fabca7de63c1cadea0a7c4937c353646cd4f32d67e61: Status 404 returned error can't find the container with id 724ce1ed1dc1d1cf91e1fabca7de63c1cadea0a7c4937c353646cd4f32d67e61
Sep 18 18:50:45.878057 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:45.877988    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerID="9c7718ebea481a2c44142ec5394ab381b7bf907737f892cbc1838eeaa012e50c" exitCode=0
Sep 18 18:50:45.878057 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:45.878030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf" event={"ID":"b9d84edd-42f7-403e-a1fb-743b4e91b329","Type":"ContainerDied","Data":"9c7718ebea481a2c44142ec5394ab381b7bf907737f892cbc1838eeaa012e50c"}
Sep 18 18:50:45.878483 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:45.878063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf" event={"ID":"b9d84edd-42f7-403e-a1fb-743b4e91b329","Type":"ContainerStarted","Data":"724ce1ed1dc1d1cf91e1fabca7de63c1cadea0a7c4937c353646cd4f32d67e61"}
Sep 18 18:50:46.883347 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:46.883293    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerID="c169febcf62012ec575947fb351d0d81726ae42ebc425ff2fbfb76a38f86db85" exitCode=0
Sep 18 18:50:46.883615 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:46.883365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf" event={"ID":"b9d84edd-42f7-403e-a1fb-743b4e91b329","Type":"ContainerDied","Data":"c169febcf62012ec575947fb351d0d81726ae42ebc425ff2fbfb76a38f86db85"}
Sep 18 18:50:47.889269 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:47.889240    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerID="7cc85a63be54a76706ecabf35c7a4666644ab77635c0c5f8c9e6b153415648ea" exitCode=0
Sep 18 18:50:47.889674 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:47.889322    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf" event={"ID":"b9d84edd-42f7-403e-a1fb-743b4e91b329","Type":"ContainerDied","Data":"7cc85a63be54a76706ecabf35c7a4666644ab77635c0c5f8c9e6b153415648ea"}
Sep 18 18:50:49.016704 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.016682    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:49.084252 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.084232    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-bundle\") pod \"b9d84edd-42f7-403e-a1fb-743b4e91b329\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") "
Sep 18 18:50:49.084392 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.084277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-util\") pod \"b9d84edd-42f7-403e-a1fb-743b4e91b329\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") "
Sep 18 18:50:49.084392 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.084314    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2qb5\" (UniqueName: \"kubernetes.io/projected/b9d84edd-42f7-403e-a1fb-743b4e91b329-kube-api-access-t2qb5\") pod \"b9d84edd-42f7-403e-a1fb-743b4e91b329\" (UID: \"b9d84edd-42f7-403e-a1fb-743b4e91b329\") "
Sep 18 18:50:49.084963 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.084937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-bundle" (OuterVolumeSpecName: "bundle") pod "b9d84edd-42f7-403e-a1fb-743b4e91b329" (UID: "b9d84edd-42f7-403e-a1fb-743b4e91b329"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:50:49.086348 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.086318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d84edd-42f7-403e-a1fb-743b4e91b329-kube-api-access-t2qb5" (OuterVolumeSpecName: "kube-api-access-t2qb5") pod "b9d84edd-42f7-403e-a1fb-743b4e91b329" (UID: "b9d84edd-42f7-403e-a1fb-743b4e91b329"). InnerVolumeSpecName "kube-api-access-t2qb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:50:49.090179 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.090140    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-util" (OuterVolumeSpecName: "util") pod "b9d84edd-42f7-403e-a1fb-743b4e91b329" (UID: "b9d84edd-42f7-403e-a1fb-743b4e91b329"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:50:49.185361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.185316    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qb5\" (UniqueName: \"kubernetes.io/projected/b9d84edd-42f7-403e-a1fb-743b4e91b329-kube-api-access-t2qb5\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:50:49.185361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.185335    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:50:49.185361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.185344    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9d84edd-42f7-403e-a1fb-743b4e91b329-util\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:50:49.897188 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.897162    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf"
Sep 18 18:50:49.897323 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.897160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkgsnf" event={"ID":"b9d84edd-42f7-403e-a1fb-743b4e91b329","Type":"ContainerDied","Data":"724ce1ed1dc1d1cf91e1fabca7de63c1cadea0a7c4937c353646cd4f32d67e61"}
Sep 18 18:50:49.897323 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:49.897274    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="724ce1ed1dc1d1cf91e1fabca7de63c1cadea0a7c4937c353646cd4f32d67e61"
Sep 18 18:50:50.449479 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.449448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:50.521688 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.521658    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"]
Sep 18 18:50:50.521973 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.521960    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerName="pull"
Sep 18 18:50:50.522019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.521975    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerName="pull"
Sep 18 18:50:50.522019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.521986    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerName="util"
Sep 18 18:50:50.522019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.521991    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerName="util"
Sep 18 18:50:50.522019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.522008    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerName="extract"
Sep 18 18:50:50.522019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.522013    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerName="extract"
Sep 18 18:50:50.522165 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.522057    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d84edd-42f7-403e-a1fb-743b4e91b329" containerName="extract"
Sep 18 18:50:50.524974 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.524959    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.528248 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.528231    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 18:50:50.528337 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.528260    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 18:50:50.528651 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.528635    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 18:50:50.528749 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.528697    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 18:50:50.528749 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.528707    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cv7vk\""
Sep 18 18:50:50.528749 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.528739    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 18:50:50.538513 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.538491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"]
Sep 18 18:50:50.549188 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.549168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:50.595155 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.595134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-cert\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.595247 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.595185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhj8t\" (UniqueName: \"kubernetes.io/projected/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-kube-api-access-nhj8t\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.595247 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.595203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-metrics-cert\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.595319 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.595270    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-manager-config\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.695774 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.695754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-cert\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.695873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.695803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhj8t\" (UniqueName: \"kubernetes.io/projected/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-kube-api-access-nhj8t\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.695873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.695820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-metrics-cert\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.695873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.695842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-manager-config\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.696401 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.696382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-manager-config\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.698211 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.698182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-cert\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.698295 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.698282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-metrics-cert\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.705122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.705068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhj8t\" (UniqueName: \"kubernetes.io/projected/1d254738-55cf-4e18-ae85-ca1fe7f0b9c3-kube-api-access-nhj8t\") pod \"lws-controller-manager-bf466d58c-7dknq\" (UID: \"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3\") " pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:50.834158 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:50.834124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:51.103370 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:51.103264    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"]
Sep 18 18:50:51.105725 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:50:51.105695    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d254738_55cf_4e18_ae85_ca1fe7f0b9c3.slice/crio-814e1a9575d3237c8358693c8940996b6e0c72c63c0236d8dd3f3820c7692690 WatchSource:0}: Error finding container 814e1a9575d3237c8358693c8940996b6e0c72c63c0236d8dd3f3820c7692690: Status 404 returned error can't find the container with id 814e1a9575d3237c8358693c8940996b6e0c72c63c0236d8dd3f3820c7692690
Sep 18 18:50:51.908151 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:51.908120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq" event={"ID":"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3","Type":"ContainerStarted","Data":"814e1a9575d3237c8358693c8940996b6e0c72c63c0236d8dd3f3820c7692690"}
Sep 18 18:50:52.628646 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.628617    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58c659b464-gf48h"]
Sep 18 18:50:52.633397 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.633374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.645436 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.645413    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c659b464-gf48h"]
Sep 18 18:50:52.705133 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.705106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:52.713345 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.713318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12557472-1266-435f-bebd-a6f3726e0ee2-console-oauth-config\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.713459 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.713360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-oauth-serving-cert\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.713522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.713464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxqt\" (UniqueName: \"kubernetes.io/projected/12557472-1266-435f-bebd-a6f3726e0ee2-kube-api-access-kpxqt\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.713522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.713492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12557472-1266-435f-bebd-a6f3726e0ee2-console-serving-cert\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.713522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.713516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-service-ca\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.713647 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.713538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-trusted-ca-bundle\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.713647 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.713562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-console-config\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.805652 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.805623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:52.814871 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.814842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12557472-1266-435f-bebd-a6f3726e0ee2-console-oauth-config\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.814998 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.814911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-oauth-serving-cert\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815045 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxqt\" (UniqueName: \"kubernetes.io/projected/12557472-1266-435f-bebd-a6f3726e0ee2-kube-api-access-kpxqt\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815045 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12557472-1266-435f-bebd-a6f3726e0ee2-console-serving-cert\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815140 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-service-ca\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815140 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-trusted-ca-bundle\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815140 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815110    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-console-config\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815730 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-oauth-serving-cert\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815730 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-service-ca\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.815981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.815955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-console-config\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.816204 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.816185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/12557472-1266-435f-bebd-a6f3726e0ee2-trusted-ca-bundle\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.817767 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.817747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12557472-1266-435f-bebd-a6f3726e0ee2-console-oauth-config\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.817959 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.817942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12557472-1266-435f-bebd-a6f3726e0ee2-console-serving-cert\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.824619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.824591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxqt\" (UniqueName: \"kubernetes.io/projected/12557472-1266-435f-bebd-a6f3726e0ee2-kube-api-access-kpxqt\") pod \"console-58c659b464-gf48h\" (UID: \"12557472-1266-435f-bebd-a6f3726e0ee2\") " pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:52.873937 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.873909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:52.943896 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:52.943859    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:50:53.423691 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.423655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:53.489403 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.489373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:50:53.819621 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.819594    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c659b464-gf48h"]
Sep 18 18:50:53.822337 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:50:53.822307    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12557472_1266_435f_bebd_a6f3726e0ee2.slice/crio-e8f0130bf520bba55ba8fde049d0026a429477e3c97f542d9d2c76ff548c759e WatchSource:0}: Error finding container e8f0130bf520bba55ba8fde049d0026a429477e3c97f542d9d2c76ff548c759e: Status 404 returned error can't find the container with id e8f0130bf520bba55ba8fde049d0026a429477e3c97f542d9d2c76ff548c759e
Sep 18 18:50:53.921694 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.921652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c659b464-gf48h" event={"ID":"12557472-1266-435f-bebd-a6f3726e0ee2","Type":"ContainerStarted","Data":"1d9680d9d20b2c68ffacba788d3c4b77702816aac1b71587a1483430de46cead"}
Sep 18 18:50:53.921841 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.921721    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c659b464-gf48h" event={"ID":"12557472-1266-435f-bebd-a6f3726e0ee2","Type":"ContainerStarted","Data":"e8f0130bf520bba55ba8fde049d0026a429477e3c97f542d9d2c76ff548c759e"}
Sep 18 18:50:53.923432 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.923410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq" event={"ID":"1d254738-55cf-4e18-ae85-ca1fe7f0b9c3","Type":"ContainerStarted","Data":"ccc382d7d832e2a8457501b0f59e97a2599605e1cf41d8af17eb192e950ad036"}
Sep 18 18:50:53.923541 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.923528    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:50:53.940383 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.940304    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58c659b464-gf48h" podStartSLOduration=1.9402906770000001 podStartE2EDuration="1.940290677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:50:53.939202825 +0000 UTC m=+377.731879779" watchObservedRunningTime="2026-09-18 18:50:53.940290677 +0000 UTC m=+377.732967606"
Sep 18 18:50:53.956796 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:50:53.956755    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq" podStartSLOduration=1.442197436 podStartE2EDuration="3.956739231s" totalImagesPullingTime="2.514541795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:50:53.955485263 +0000 UTC m=+377.748162210" watchObservedRunningTime="2026-09-18 18:50:53.956739231 +0000 UTC m=+377.749416161"
Sep 18 18:51:00.725942 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.725911    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"]
Sep 18 18:51:00.729380 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.729363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.733045 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.733022    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 18:51:00.733185 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.733031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mxnzd\""
Sep 18 18:51:00.733702 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.733688    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 18:51:00.743668 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.743647    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"]
Sep 18 18:51:00.874857 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.874832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.874990 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.874871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.874990 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.874946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkpd\" (UniqueName: \"kubernetes.io/projected/aae1d9b8-5232-44e2-9adb-27eef2c226d5-kube-api-access-8vkpd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.976103 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.976051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkpd\" (UniqueName: \"kubernetes.io/projected/aae1d9b8-5232-44e2-9adb-27eef2c226d5-kube-api-access-8vkpd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.976103 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.976092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.976219 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.976121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.976410 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.976395    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.976452 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.976439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:00.984854 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:00.984836    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkpd\" (UniqueName: \"kubernetes.io/projected/aae1d9b8-5232-44e2-9adb-27eef2c226d5-kube-api-access-8vkpd\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:01.037616 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:01.037595    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:01.086356 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:01.086296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:01.187506 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:01.187477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:01.310112 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:01.310066    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"]
Sep 18 18:51:01.312343 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:51:01.312317    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae1d9b8_5232_44e2_9adb_27eef2c226d5.slice/crio-12ecc641a27e990639baf146fae51aec8f6af2f6df035b5a219c1e15f1876bd4 WatchSource:0}: Error finding container 12ecc641a27e990639baf146fae51aec8f6af2f6df035b5a219c1e15f1876bd4: Status 404 returned error can't find the container with id 12ecc641a27e990639baf146fae51aec8f6af2f6df035b5a219c1e15f1876bd4
Sep 18 18:51:01.960175 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:01.960139    2582 generic.go:356] "Generic (PLEG): container finished" podID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerID="f6953170e27d15b0992840722120dd71190442d2f4da2d640d624e63f18f740a" exitCode=0
Sep 18 18:51:01.960467 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:01.960197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw" event={"ID":"aae1d9b8-5232-44e2-9adb-27eef2c226d5","Type":"ContainerDied","Data":"f6953170e27d15b0992840722120dd71190442d2f4da2d640d624e63f18f740a"}
Sep 18 18:51:01.960467 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:01.960234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw" event={"ID":"aae1d9b8-5232-44e2-9adb-27eef2c226d5","Type":"ContainerStarted","Data":"12ecc641a27e990639baf146fae51aec8f6af2f6df035b5a219c1e15f1876bd4"}
Sep 18 18:51:02.026573 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.026547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:02.555698 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.555671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:02.651284 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.651263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:02.718745 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.718720    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"]
Sep 18 18:51:02.721876 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.721862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.725924 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.725907    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 18:51:02.726021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.726005    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 18:51:02.726075 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.726006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 18:51:02.726184 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.726168    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t6jkl\""
Sep 18 18:51:02.731699 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.731684    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 18:51:02.750279 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.750256    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"]
Sep 18 18:51:02.791742 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.791693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10789836-47ae-4a4b-830c-4eb9f7d4616c-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.791742 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.791728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10789836-47ae-4a4b-830c-4eb9f7d4616c-manifests\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.791919 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.791747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10789836-47ae-4a4b-830c-4eb9f7d4616c-tmp\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.791919 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.791765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10789836-47ae-4a4b-830c-4eb9f7d4616c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.792024 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.791930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp8mm\" (UniqueName: \"kubernetes.io/projected/10789836-47ae-4a4b-830c-4eb9f7d4616c-kube-api-access-dp8mm\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.892601 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.892576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp8mm\" (UniqueName: \"kubernetes.io/projected/10789836-47ae-4a4b-830c-4eb9f7d4616c-kube-api-access-dp8mm\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.892688 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.892610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10789836-47ae-4a4b-830c-4eb9f7d4616c-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.892688 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.892635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10789836-47ae-4a4b-830c-4eb9f7d4616c-manifests\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.892688 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.892653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10789836-47ae-4a4b-830c-4eb9f7d4616c-tmp\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.892688 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.892679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10789836-47ae-4a4b-830c-4eb9f7d4616c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.893031 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.893012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10789836-47ae-4a4b-830c-4eb9f7d4616c-manifests\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.894992 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.894969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10789836-47ae-4a4b-830c-4eb9f7d4616c-tmp\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.895200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.895183    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10789836-47ae-4a4b-830c-4eb9f7d4616c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.895312 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.895294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10789836-47ae-4a4b-830c-4eb9f7d4616c-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.902229 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.902211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp8mm\" (UniqueName: \"kubernetes.io/projected/10789836-47ae-4a4b-830c-4eb9f7d4616c-kube-api-access-dp8mm\") pod \"opendatahub-operator-controller-manager-699fb7c78d-dsqd4\" (UID: \"10789836-47ae-4a4b-830c-4eb9f7d4616c\") " pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:02.944504 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.944481    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:51:02.944573 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.944531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:51:02.949539 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.949520    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:51:02.966377 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.966357    2582 generic.go:356] "Generic (PLEG): container finished" podID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerID="5a1846ad9d7c97abfc3aa7737fe0ca84401d8f08ef13651b14df43c64e2c256d" exitCode=0
Sep 18 18:51:02.966692 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.966441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw" event={"ID":"aae1d9b8-5232-44e2-9adb-27eef2c226d5","Type":"ContainerDied","Data":"5a1846ad9d7c97abfc3aa7737fe0ca84401d8f08ef13651b14df43c64e2c256d"}
Sep 18 18:51:02.968451 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.968430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:02.970314 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:02.970291    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58c659b464-gf48h"
Sep 18 18:51:03.031006 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.030983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:03.051784 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.051753    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85bf49897c-7lzx4"]
Sep 18 18:51:03.102269 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.102134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:03.241743 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.241720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:03.326419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.326393    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"]
Sep 18 18:51:03.327256 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:51:03.327234    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10789836_47ae_4a4b_830c_4eb9f7d4616c.slice/crio-cdeb2f074a5052ca1438b50347e4b40e7011a4a2ab4f543341c6affffa4d20b9 WatchSource:0}: Error finding container cdeb2f074a5052ca1438b50347e4b40e7011a4a2ab4f543341c6affffa4d20b9: Status 404 returned error can't find the container with id cdeb2f074a5052ca1438b50347e4b40e7011a4a2ab4f543341c6affffa4d20b9
Sep 18 18:51:03.352659 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.352639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:03.448826 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.448803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:03.553109 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.553086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:03.972685 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.972657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4" event={"ID":"10789836-47ae-4a4b-830c-4eb9f7d4616c","Type":"ContainerStarted","Data":"cdeb2f074a5052ca1438b50347e4b40e7011a4a2ab4f543341c6affffa4d20b9"}
Sep 18 18:51:03.974753 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.974722    2582 generic.go:356] "Generic (PLEG): container finished" podID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerID="8e20b8ad3657b81a609762535295e0a88262da907fb24aa9909d33be75ef7a6d" exitCode=0
Sep 18 18:51:03.974849 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:03.974803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw" event={"ID":"aae1d9b8-5232-44e2-9adb-27eef2c226d5","Type":"ContainerDied","Data":"8e20b8ad3657b81a609762535295e0a88262da907fb24aa9909d33be75ef7a6d"}
Sep 18 18:51:04.935822 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:04.935790    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-bf466d58c-7dknq"
Sep 18 18:51:06.290766 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.290744    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:06.423652 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.423630    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-util\") pod \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") "
Sep 18 18:51:06.423740 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.423687    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-bundle\") pod \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") "
Sep 18 18:51:06.423740 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.423731    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vkpd\" (UniqueName: \"kubernetes.io/projected/aae1d9b8-5232-44e2-9adb-27eef2c226d5-kube-api-access-8vkpd\") pod \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\" (UID: \"aae1d9b8-5232-44e2-9adb-27eef2c226d5\") "
Sep 18 18:51:06.424435 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.424348    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-bundle" (OuterVolumeSpecName: "bundle") pod "aae1d9b8-5232-44e2-9adb-27eef2c226d5" (UID: "aae1d9b8-5232-44e2-9adb-27eef2c226d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:51:06.425630 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.425602    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae1d9b8-5232-44e2-9adb-27eef2c226d5-kube-api-access-8vkpd" (OuterVolumeSpecName: "kube-api-access-8vkpd") pod "aae1d9b8-5232-44e2-9adb-27eef2c226d5" (UID: "aae1d9b8-5232-44e2-9adb-27eef2c226d5"). InnerVolumeSpecName "kube-api-access-8vkpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:51:06.428709 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.428689    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-util" (OuterVolumeSpecName: "util") pod "aae1d9b8-5232-44e2-9adb-27eef2c226d5" (UID: "aae1d9b8-5232-44e2-9adb-27eef2c226d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 18:51:06.525057 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.525013    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:06.525057 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.525033    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vkpd\" (UniqueName: \"kubernetes.io/projected/aae1d9b8-5232-44e2-9adb-27eef2c226d5-kube-api-access-8vkpd\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:06.525057 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.525041    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aae1d9b8-5232-44e2-9adb-27eef2c226d5-util\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:06.992541 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.992502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4" event={"ID":"10789836-47ae-4a4b-830c-4eb9f7d4616c","Type":"ContainerStarted","Data":"10ac5687e91b5f954f1e61780d1675a846100527c0ce4cf33a67a2363a698c46"}
Sep 18 18:51:06.994339 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.994317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw" event={"ID":"aae1d9b8-5232-44e2-9adb-27eef2c226d5","Type":"ContainerDied","Data":"12ecc641a27e990639baf146fae51aec8f6af2f6df035b5a219c1e15f1876bd4"}
Sep 18 18:51:06.994423 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.994352    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12ecc641a27e990639baf146fae51aec8f6af2f6df035b5a219c1e15f1876bd4"
Sep 18 18:51:06.994423 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:06.994354    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b049fzw"
Sep 18 18:51:08.000281 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:08.000257    2582 generic.go:356] "Generic (PLEG): container finished" podID="10789836-47ae-4a4b-830c-4eb9f7d4616c" containerID="10ac5687e91b5f954f1e61780d1675a846100527c0ce4cf33a67a2363a698c46" exitCode=0
Sep 18 18:51:08.000600 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:08.000350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4" event={"ID":"10789836-47ae-4a4b-830c-4eb9f7d4616c","Type":"ContainerDied","Data":"10ac5687e91b5f954f1e61780d1675a846100527c0ce4cf33a67a2363a698c46"}
Sep 18 18:51:09.007528 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:09.007499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4" event={"ID":"10789836-47ae-4a4b-830c-4eb9f7d4616c","Type":"ContainerStarted","Data":"11309edef5ce7d4372953dbf8a6e1184e5610dfee1604ebed86f5ef7b85e47ba"}
Sep 18 18:51:09.007858 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:09.007613    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:09.032646 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:09.032598    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4" podStartSLOduration=2.689852623 podStartE2EDuration="7.032581016s" totalImagesPullingTime="3.379845085s" totalInitContainerRuntime="962.883308ms" isStatefulPod=false podCreationTimestamp="2026-09-18 18:51:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:51:09.031216471 +0000 UTC m=+392.823893416" watchObservedRunningTime="2026-09-18 18:51:09.032581016 +0000 UTC m=+392.825257946"
Sep 18 18:51:20.013314 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:20.013273    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-699fb7c78d-dsqd4"
Sep 18 18:51:22.078172 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:22.078132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:22.246574 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:22.246551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:23.976680 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:23.976649    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:24.030360 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030331    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"]
Sep 18 18:51:24.030649 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030636    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerName="pull"
Sep 18 18:51:24.030721 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030657    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerName="pull"
Sep 18 18:51:24.030721 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030669    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerName="util"
Sep 18 18:51:24.030721 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030678    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerName="util"
Sep 18 18:51:24.030721 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030700    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerName="extract"
Sep 18 18:51:24.030721 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030705    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerName="extract"
Sep 18 18:51:24.030981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.030762    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="aae1d9b8-5232-44e2-9adb-27eef2c226d5" containerName="extract"
Sep 18 18:51:24.033353 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.033332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.035776 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.035756    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 18:51:24.035776 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.035770    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v2zvp\""
Sep 18 18:51:24.036562 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.036539    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 18:51:24.036625 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.036581    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 18:51:24.036757 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.036730    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 18:51:24.046278 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.046255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"]
Sep 18 18:51:24.144598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.144572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695dbc20-cce1-45f9-81c3-29e34b936f4d-tls-certs\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.144598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.144598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/695dbc20-cce1-45f9-81c3-29e34b936f4d-tmp\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.144735 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.144625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hgr\" (UniqueName: \"kubernetes.io/projected/695dbc20-cce1-45f9-81c3-29e34b936f4d-kube-api-access-q7hgr\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.245665 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.245613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hgr\" (UniqueName: \"kubernetes.io/projected/695dbc20-cce1-45f9-81c3-29e34b936f4d-kube-api-access-q7hgr\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.245747 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.245673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695dbc20-cce1-45f9-81c3-29e34b936f4d-tls-certs\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.245747 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.245695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/695dbc20-cce1-45f9-81c3-29e34b936f4d-tmp\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.248073 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.248046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/695dbc20-cce1-45f9-81c3-29e34b936f4d-tmp\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.248250 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.248235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695dbc20-cce1-45f9-81c3-29e34b936f4d-tls-certs\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.264045 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.264025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hgr\" (UniqueName: \"kubernetes.io/projected/695dbc20-cce1-45f9-81c3-29e34b936f4d-kube-api-access-q7hgr\") pod \"kube-auth-proxy-6cd84dd4ff-8zn5r\" (UID: \"695dbc20-cce1-45f9-81c3-29e34b936f4d\") " pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.284186 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.284155    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:24.343248 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.343230    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"
Sep 18 18:51:24.610761 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:24.610733    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r"]
Sep 18 18:51:24.612117 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:51:24.612096    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695dbc20_cce1_45f9_81c3_29e34b936f4d.slice/crio-ccde70e5e66a4f0331c5d8dd994fb290805f3254748ab736dc3f567597223f23 WatchSource:0}: Error finding container ccde70e5e66a4f0331c5d8dd994fb290805f3254748ab736dc3f567597223f23: Status 404 returned error can't find the container with id ccde70e5e66a4f0331c5d8dd994fb290805f3254748ab736dc3f567597223f23
Sep 18 18:51:25.074965 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:25.074930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r" event={"ID":"695dbc20-cce1-45f9-81c3-29e34b936f4d","Type":"ContainerStarted","Data":"ccde70e5e66a4f0331c5d8dd994fb290805f3254748ab736dc3f567597223f23"}
Sep 18 18:51:25.481313 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:25.481243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:25.574351 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:25.574321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:26.758069 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:26.758034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:28.073811 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.073727    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85bf49897c-7lzx4" podUID="513ee270-d8fd-42a4-aae0-f47a2be252fc" containerName="console" containerID="cri-o://744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f" gracePeriod=15
Sep 18 18:51:28.091577 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.091547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r" event={"ID":"695dbc20-cce1-45f9-81c3-29e34b936f4d","Type":"ContainerStarted","Data":"6827c62149d85e5764312910b8b8153e3c16e2485fe3fd8d964918b10446dde8"}
Sep 18 18:51:28.125460 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.125419    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6cd84dd4ff-8zn5r" podStartSLOduration=1.049868799 podStartE2EDuration="4.125404516s" totalImagesPullingTime="3.075535717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:51:28.122636286 +0000 UTC m=+411.915313215" watchObservedRunningTime="2026-09-18 18:51:28.125404516 +0000 UTC m=+411.918081469"
Sep 18 18:51:28.319939 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.319915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:51:28.482181 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482155    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-config\") pod \"513ee270-d8fd-42a4-aae0-f47a2be252fc\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") "
Sep 18 18:51:28.482327 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-oauth-config\") pod \"513ee270-d8fd-42a4-aae0-f47a2be252fc\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") "
Sep 18 18:51:28.482327 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482222    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-trusted-ca-bundle\") pod \"513ee270-d8fd-42a4-aae0-f47a2be252fc\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") "
Sep 18 18:51:28.482327 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482244    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnxrq\" (UniqueName: \"kubernetes.io/projected/513ee270-d8fd-42a4-aae0-f47a2be252fc-kube-api-access-fnxrq\") pod \"513ee270-d8fd-42a4-aae0-f47a2be252fc\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") "
Sep 18 18:51:28.482327 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482266    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-oauth-serving-cert\") pod \"513ee270-d8fd-42a4-aae0-f47a2be252fc\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") "
Sep 18 18:51:28.482327 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482314    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-serving-cert\") pod \"513ee270-d8fd-42a4-aae0-f47a2be252fc\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") "
Sep 18 18:51:28.482582 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-service-ca\") pod \"513ee270-d8fd-42a4-aae0-f47a2be252fc\" (UID: \"513ee270-d8fd-42a4-aae0-f47a2be252fc\") "
Sep 18 18:51:28.482708 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-config" (OuterVolumeSpecName: "console-config") pod "513ee270-d8fd-42a4-aae0-f47a2be252fc" (UID: "513ee270-d8fd-42a4-aae0-f47a2be252fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:51:28.482708 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482698    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "513ee270-d8fd-42a4-aae0-f47a2be252fc" (UID: "513ee270-d8fd-42a4-aae0-f47a2be252fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:51:28.482861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "513ee270-d8fd-42a4-aae0-f47a2be252fc" (UID: "513ee270-d8fd-42a4-aae0-f47a2be252fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:51:28.482861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.482805    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "513ee270-d8fd-42a4-aae0-f47a2be252fc" (UID: "513ee270-d8fd-42a4-aae0-f47a2be252fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:51:28.484527 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.484506    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "513ee270-d8fd-42a4-aae0-f47a2be252fc" (UID: "513ee270-d8fd-42a4-aae0-f47a2be252fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:51:28.484605 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.484550    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "513ee270-d8fd-42a4-aae0-f47a2be252fc" (UID: "513ee270-d8fd-42a4-aae0-f47a2be252fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:51:28.484648 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.484593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513ee270-d8fd-42a4-aae0-f47a2be252fc-kube-api-access-fnxrq" (OuterVolumeSpecName: "kube-api-access-fnxrq") pod "513ee270-d8fd-42a4-aae0-f47a2be252fc" (UID: "513ee270-d8fd-42a4-aae0-f47a2be252fc"). InnerVolumeSpecName "kube-api-access-fnxrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:51:28.583283 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.583262    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-oauth-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:28.583283 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.583282    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-serving-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:28.583382 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.583290    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:28.583382 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.583299    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:28.583382 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.583306    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/513ee270-d8fd-42a4-aae0-f47a2be252fc-console-oauth-config\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:28.583382 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.583314    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/513ee270-d8fd-42a4-aae0-f47a2be252fc-trusted-ca-bundle\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:28.583382 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:28.583322    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxrq\" (UniqueName: \"kubernetes.io/projected/513ee270-d8fd-42a4-aae0-f47a2be252fc-kube-api-access-fnxrq\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:51:29.097307 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.097272    2582 generic.go:356] "Generic (PLEG): container finished" podID="513ee270-d8fd-42a4-aae0-f47a2be252fc" containerID="744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f" exitCode=0
Sep 18 18:51:29.097613 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.097315    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85bf49897c-7lzx4"
Sep 18 18:51:29.097613 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.097356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bf49897c-7lzx4" event={"ID":"513ee270-d8fd-42a4-aae0-f47a2be252fc","Type":"ContainerDied","Data":"744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f"}
Sep 18 18:51:29.097613 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.097405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85bf49897c-7lzx4" event={"ID":"513ee270-d8fd-42a4-aae0-f47a2be252fc","Type":"ContainerDied","Data":"3599858599b4decefd37f34c0f8c816ee9a363cf7bfbd113cef2fb54971151e0"}
Sep 18 18:51:29.097613 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.097431    2582 scope.go:117] "RemoveContainer" containerID="744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f"
Sep 18 18:51:29.106336 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.106321    2582 scope.go:117] "RemoveContainer" containerID="744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f"
Sep 18 18:51:29.106584 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:51:29.106567    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f\": container with ID starting with 744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f not found: ID does not exist" containerID="744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f"
Sep 18 18:51:29.106631 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.106599    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f"} err="failed to get container status \"744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f\": rpc error: code = NotFound desc = could not find container \"744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f\": container with ID starting with 744b104943a1108c02c08d452fe4ec45fa51181c42ec3d0d18c76730386d6d0f not found: ID does not exist"
Sep 18 18:51:29.119405 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.119380    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85bf49897c-7lzx4"]
Sep 18 18:51:29.123304 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:29.123284    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85bf49897c-7lzx4"]
Sep 18 18:51:30.178225 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:30.178200    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:30.278632 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:30.278609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:30.334279 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:30.334259    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513ee270-d8fd-42a4-aae0-f47a2be252fc" path="/var/lib/kubelet/pods/513ee270-d8fd-42a4-aae0-f47a2be252fc/volumes"
Sep 18 18:51:30.490313 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:30.490250    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:30.701697 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:30.701671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:30.803197 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:30.803151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:33.191066 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:33.191036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:33.297025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:33.296996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:33.391860 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:33.391839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:33.492073 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:33.492022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:33.775040 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:33.774977    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:34.038274 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:34.038221    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:43.837431 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:43.837397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:43.959245 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:43.959211    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:44.260925 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:44.260870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:44.383512 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:44.383485    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:45.037970 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:45.037939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:45.162029 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:45.161999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:45.255930 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:45.255903    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:51:45.355000 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:51:45.354942    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:11.278227 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:11.278153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:34.183361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:34.183325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:34.388279 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:34.388247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:53.257808 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:53.257775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:54.049162 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:54.049129    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:54.839316 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:54.839271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:56.957395 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:56.957359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:57.069632 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:57.069604    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:52:58.332109 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:52:58.332071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:00.366391 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:00.366357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:00.621425 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:00.621357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:02.541345 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:02.541314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:02.976073 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:02.976052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:03.207610 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:03.207584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:05.507728 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:05.507703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:05.696506 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:05.696476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:08.355066 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:08.355032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:08.488774 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:08.488744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:08.997361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:08.997333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:09.032936 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.032908    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"]
Sep 18 18:53:09.033207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.033196    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513ee270-d8fd-42a4-aae0-f47a2be252fc" containerName="console"
Sep 18 18:53:09.033270 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.033209    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="513ee270-d8fd-42a4-aae0-f47a2be252fc" containerName="console"
Sep 18 18:53:09.033303 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.033275    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="513ee270-d8fd-42a4-aae0-f47a2be252fc" containerName="console"
Sep 18 18:53:09.036053 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.036038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.038749 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.038733    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ldxjs\""
Sep 18 18:53:09.039370 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.039355    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 18:53:09.041552 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.041531    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 18:53:09.041642 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.041598    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 18:53:09.049605 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.049589    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 18:53:09.061785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.061764    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"]
Sep 18 18:53:09.105402 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.105380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.105488 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.105415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.105534 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.105492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhq8\" (UniqueName: \"kubernetes.io/projected/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-kube-api-access-2zhq8\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.198479 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.198456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:09.206567 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.206550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhq8\" (UniqueName: \"kubernetes.io/projected/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-kube-api-access-2zhq8\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.206633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.206581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.206633 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.206606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.207131 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.207114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.209069 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.209049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.222415 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.222398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhq8\" (UniqueName: \"kubernetes.io/projected/3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd-kube-api-access-2zhq8\") pod \"kuadrant-console-plugin-fbdd4f445-dkn2b\" (UID: \"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.345209 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.345160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"
Sep 18 18:53:09.371109 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.371088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:09.517941 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.517911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:09.626849 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:09.626829    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b"]
Sep 18 18:53:09.628842 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:53:09.628805    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e8fb66d_99ed_4edd_b3cc_65afb3fed7dd.slice/crio-384136e7c4357f596c913c5dfe3e83ab0bf4b0eb06e2a9852bc974b6c398d2a1 WatchSource:0}: Error finding container 384136e7c4357f596c913c5dfe3e83ab0bf4b0eb06e2a9852bc974b6c398d2a1: Status 404 returned error can't find the container with id 384136e7c4357f596c913c5dfe3e83ab0bf4b0eb06e2a9852bc974b6c398d2a1
Sep 18 18:53:10.508874 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:10.508844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b" event={"ID":"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd","Type":"ContainerStarted","Data":"384136e7c4357f596c913c5dfe3e83ab0bf4b0eb06e2a9852bc974b6c398d2a1"}
Sep 18 18:53:15.534632 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:15.534600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b" event={"ID":"3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd","Type":"ContainerStarted","Data":"15f71b21df33faedab97b10f825c860cf35c17d76f31a717e7e7389f900dc448"}
Sep 18 18:53:16.545393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:16.545348    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:16.798095 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:16.798035    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:19.025981 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:19.025902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:19.332840 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:19.332789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:25.422393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:25.422357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:25.588967 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:25.588937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:25.745152 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:25.745095    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:30.465241 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:30.465209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:30.641231 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:30.641202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:30.776153 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:30.776099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:30.945773 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:30.945749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:31.168062 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:31.168041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:31.299507 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:31.299485    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:33.158759 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:33.158732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:33.445435 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:33.445376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:37.509527 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:37.509493    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:37.650631 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:37.650604    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:40.233831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:40.233800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:40.413317 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:40.413293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:46.483877 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:46.483842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:46.614824 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:46.614774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:51.659242 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:51.659216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:51.799308 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:51.799281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:52.079406 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:52.079344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:52.190488 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:52.190468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:52.297637 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:52.297613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:52.407565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:52.407545    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:53.397952 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:53.397923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:53.542429 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:53.542408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:59.012407 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.012372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:59.131831 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.131803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:59.224807 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.224763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:59.493102 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.493072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:59.529684 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.529638    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-dkn2b" podStartSLOduration=46.503113409 podStartE2EDuration="51.529624491s" totalImagesPullingTime="5.026511082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:53:15.603319362 +0000 UTC m=+519.395996293" watchObservedRunningTime="2026-09-18 18:53:59.529624491 +0000 UTC m=+563.322301419"
Sep 18 18:53:59.530010 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.529841    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:53:59.533214 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.533196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:53:59.537621 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.537604    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 18:53:59.549461 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.549440    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:53:59.570854 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.570832    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:53:59.614516 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.614493    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:53:59.658018 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.658000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7mg7\" (UniqueName: \"kubernetes.io/projected/660c1cf1-044e-444f-a156-607398e93597-kube-api-access-s7mg7\") pod \"limitador-limitador-84994f7ffc-5zqwn\" (UID: \"660c1cf1-044e-444f-a156-607398e93597\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:53:59.658122 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.658027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/660c1cf1-044e-444f-a156-607398e93597-config-file\") pod \"limitador-limitador-84994f7ffc-5zqwn\" (UID: \"660c1cf1-044e-444f-a156-607398e93597\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:53:59.758593 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.758533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7mg7\" (UniqueName: \"kubernetes.io/projected/660c1cf1-044e-444f-a156-607398e93597-kube-api-access-s7mg7\") pod \"limitador-limitador-84994f7ffc-5zqwn\" (UID: \"660c1cf1-044e-444f-a156-607398e93597\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:53:59.758593 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.758563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/660c1cf1-044e-444f-a156-607398e93597-config-file\") pod \"limitador-limitador-84994f7ffc-5zqwn\" (UID: \"660c1cf1-044e-444f-a156-607398e93597\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:53:59.759102 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.759086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/660c1cf1-044e-444f-a156-607398e93597-config-file\") pod \"limitador-limitador-84994f7ffc-5zqwn\" (UID: \"660c1cf1-044e-444f-a156-607398e93597\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:53:59.767523 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.767497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7mg7\" (UniqueName: \"kubernetes.io/projected/660c1cf1-044e-444f-a156-607398e93597-kube-api-access-s7mg7\") pod \"limitador-limitador-84994f7ffc-5zqwn\" (UID: \"660c1cf1-044e-444f-a156-607398e93597\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:53:59.842974 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:53:59.842956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:54:00.028605 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.028535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:00.118785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.118758    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:54:00.119993 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:54:00.119971    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660c1cf1_044e_444f_a156_607398e93597.slice/crio-ab6ccf218da1c55128e606cb65c6e88f5c5c2c7819a0c29835f61c9bf36057a4 WatchSource:0}: Error finding container ab6ccf218da1c55128e606cb65c6e88f5c5c2c7819a0c29835f61c9bf36057a4: Status 404 returned error can't find the container with id ab6ccf218da1c55128e606cb65c6e88f5c5c2c7819a0c29835f61c9bf36057a4
Sep 18 18:54:00.142034 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.142014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:00.239686 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.239662    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2s7rs"]
Sep 18 18:54:00.244041 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.244026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2s7rs"
Sep 18 18:54:00.246741 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.246722    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wtn2g\""
Sep 18 18:54:00.253756 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.253737    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2s7rs"]
Sep 18 18:54:00.363911 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.363844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch4fb\" (UniqueName: \"kubernetes.io/projected/fb4be7cd-2811-4abd-ae21-fcb7baf2b153-kube-api-access-ch4fb\") pod \"authorino-59c974bd48-2s7rs\" (UID: \"fb4be7cd-2811-4abd-ae21-fcb7baf2b153\") " pod="kuadrant-system/authorino-59c974bd48-2s7rs"
Sep 18 18:54:00.465127 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.465108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch4fb\" (UniqueName: \"kubernetes.io/projected/fb4be7cd-2811-4abd-ae21-fcb7baf2b153-kube-api-access-ch4fb\") pod \"authorino-59c974bd48-2s7rs\" (UID: \"fb4be7cd-2811-4abd-ae21-fcb7baf2b153\") " pod="kuadrant-system/authorino-59c974bd48-2s7rs"
Sep 18 18:54:00.485451 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.485428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch4fb\" (UniqueName: \"kubernetes.io/projected/fb4be7cd-2811-4abd-ae21-fcb7baf2b153-kube-api-access-ch4fb\") pod \"authorino-59c974bd48-2s7rs\" (UID: \"fb4be7cd-2811-4abd-ae21-fcb7baf2b153\") " pod="kuadrant-system/authorino-59c974bd48-2s7rs"
Sep 18 18:54:00.552739 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.552718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2s7rs"
Sep 18 18:54:00.732291 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.732261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn" event={"ID":"660c1cf1-044e-444f-a156-607398e93597","Type":"ContainerStarted","Data":"ab6ccf218da1c55128e606cb65c6e88f5c5c2c7819a0c29835f61c9bf36057a4"}
Sep 18 18:54:00.826203 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:00.826179    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2s7rs"]
Sep 18 18:54:00.828510 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:54:00.828486    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4be7cd_2811_4abd_ae21_fcb7baf2b153.slice/crio-c75b9cbe1b48b3e19db3c963a3a1cc3e86d4920a04d60a7324d2e4fbede8550d WatchSource:0}: Error finding container c75b9cbe1b48b3e19db3c963a3a1cc3e86d4920a04d60a7324d2e4fbede8550d: Status 404 returned error can't find the container with id c75b9cbe1b48b3e19db3c963a3a1cc3e86d4920a04d60a7324d2e4fbede8550d
Sep 18 18:54:01.741254 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:01.741210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2s7rs" event={"ID":"fb4be7cd-2811-4abd-ae21-fcb7baf2b153","Type":"ContainerStarted","Data":"c75b9cbe1b48b3e19db3c963a3a1cc3e86d4920a04d60a7324d2e4fbede8550d"}
Sep 18 18:54:02.466945 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:02.466901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:02.563259 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:02.563233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:03.753353 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:03.753318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn" event={"ID":"660c1cf1-044e-444f-a156-607398e93597","Type":"ContainerStarted","Data":"a65dd72812e2798507399c33e96871df37f2d1c6247b30f54b8d548eddc23bff"}
Sep 18 18:54:03.753785 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:03.753473    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:54:03.754863 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:03.754841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2s7rs" event={"ID":"fb4be7cd-2811-4abd-ae21-fcb7baf2b153","Type":"ContainerStarted","Data":"2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4"}
Sep 18 18:54:03.773660 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:03.773621    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn" podStartSLOduration=1.860869793 podStartE2EDuration="4.773608767s" totalImagesPullingTime="2.912738974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:03.771685158 +0000 UTC m=+567.564362092" watchObservedRunningTime="2026-09-18 18:54:03.773608767 +0000 UTC m=+567.566285697"
Sep 18 18:54:03.787783 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:03.787743    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2s7rs" podStartSLOduration=1.576917527 podStartE2EDuration="3.787729174s" totalImagesPullingTime="2.210811647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:03.786959427 +0000 UTC m=+567.579636358" watchObservedRunningTime="2026-09-18 18:54:03.787729174 +0000 UTC m=+567.580406103"
Sep 18 18:54:08.449530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:08.449500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:08.512388 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:08.512361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:09.059635 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:09.059612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:09.125413 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:09.125390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:14.759717 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:14.759690    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"
Sep 18 18:54:20.185096 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:20.185062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:20.282115 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:20.282084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:20.748744 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:20.748717    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:33.333628 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.333597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:33.573675 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.573649    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-hzt8x"]
Sep 18 18:54:33.577927 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.577909    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-hzt8x"
Sep 18 18:54:33.583775 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.583676    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-hzt8x"]
Sep 18 18:54:33.676492 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.676465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:33.704737 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.704711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hcn\" (UniqueName: \"kubernetes.io/projected/53cc878c-3ef5-4774-9312-61602ce4816c-kube-api-access-66hcn\") pod \"authorino-557f876f69-hzt8x\" (UID: \"53cc878c-3ef5-4774-9312-61602ce4816c\") " pod="kuadrant-system/authorino-557f876f69-hzt8x"
Sep 18 18:54:33.805358 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.805333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hcn\" (UniqueName: \"kubernetes.io/projected/53cc878c-3ef5-4774-9312-61602ce4816c-kube-api-access-66hcn\") pod \"authorino-557f876f69-hzt8x\" (UID: \"53cc878c-3ef5-4774-9312-61602ce4816c\") " pod="kuadrant-system/authorino-557f876f69-hzt8x"
Sep 18 18:54:33.814421 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.814399    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-hzt8x"]
Sep 18 18:54:33.814578 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:54:33.814562    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-66hcn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-hzt8x" podUID="53cc878c-3ef5-4774-9312-61602ce4816c"
Sep 18 18:54:33.816647 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.816619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hcn\" (UniqueName: \"kubernetes.io/projected/53cc878c-3ef5-4774-9312-61602ce4816c-kube-api-access-66hcn\") pod \"authorino-557f876f69-hzt8x\" (UID: \"53cc878c-3ef5-4774-9312-61602ce4816c\") " pod="kuadrant-system/authorino-557f876f69-hzt8x"
Sep 18 18:54:33.882580 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.882527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-hzt8x"
Sep 18 18:54:33.888117 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.888099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-hzt8x"
Sep 18 18:54:33.923551 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.923527    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5558864f74-hkh79"]
Sep 18 18:54:33.925822 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.925805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:33.927860 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.927845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 18:54:33.937061 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:33.937042    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5558864f74-hkh79"]
Sep 18 18:54:34.006021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.005994    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66hcn\" (UniqueName: \"kubernetes.io/projected/53cc878c-3ef5-4774-9312-61602ce4816c-kube-api-access-66hcn\") pod \"53cc878c-3ef5-4774-9312-61602ce4816c\" (UID: \"53cc878c-3ef5-4774-9312-61602ce4816c\") "
Sep 18 18:54:34.006126 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.006106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrsb7\" (UniqueName: \"kubernetes.io/projected/796fe68c-8a7a-4942-82c0-24f6b5277345-kube-api-access-wrsb7\") pod \"authorino-5558864f74-hkh79\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") " pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:34.006189 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.006151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/796fe68c-8a7a-4942-82c0-24f6b5277345-tls-cert\") pod \"authorino-5558864f74-hkh79\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") " pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:34.008104 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.008074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cc878c-3ef5-4774-9312-61602ce4816c-kube-api-access-66hcn" (OuterVolumeSpecName: "kube-api-access-66hcn") pod "53cc878c-3ef5-4774-9312-61602ce4816c" (UID: "53cc878c-3ef5-4774-9312-61602ce4816c"). InnerVolumeSpecName "kube-api-access-66hcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:54:34.107056 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.107034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrsb7\" (UniqueName: \"kubernetes.io/projected/796fe68c-8a7a-4942-82c0-24f6b5277345-kube-api-access-wrsb7\") pod \"authorino-5558864f74-hkh79\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") " pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:34.107141 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.107069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/796fe68c-8a7a-4942-82c0-24f6b5277345-tls-cert\") pod \"authorino-5558864f74-hkh79\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") " pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:34.107141 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.107125    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66hcn\" (UniqueName: \"kubernetes.io/projected/53cc878c-3ef5-4774-9312-61602ce4816c-kube-api-access-66hcn\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:54:34.109454 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.109434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/796fe68c-8a7a-4942-82c0-24f6b5277345-tls-cert\") pod \"authorino-5558864f74-hkh79\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") " pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:34.115302 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.115273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrsb7\" (UniqueName: \"kubernetes.io/projected/796fe68c-8a7a-4942-82c0-24f6b5277345-kube-api-access-wrsb7\") pod \"authorino-5558864f74-hkh79\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") " pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:34.235556 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.235537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:54:34.507310 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.507242    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5558864f74-hkh79"]
Sep 18 18:54:34.510317 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:54:34.510291    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod796fe68c_8a7a_4942_82c0_24f6b5277345.slice/crio-25fcbcaedfff0c2b97def42b72e908ed49e843cba84b6000714ccbcd1a4d3227 WatchSource:0}: Error finding container 25fcbcaedfff0c2b97def42b72e908ed49e843cba84b6000714ccbcd1a4d3227: Status 404 returned error can't find the container with id 25fcbcaedfff0c2b97def42b72e908ed49e843cba84b6000714ccbcd1a4d3227
Sep 18 18:54:34.888078 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.888026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-hzt8x"
Sep 18 18:54:34.888078 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.888039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5558864f74-hkh79" event={"ID":"796fe68c-8a7a-4942-82c0-24f6b5277345","Type":"ContainerStarted","Data":"25fcbcaedfff0c2b97def42b72e908ed49e843cba84b6000714ccbcd1a4d3227"}
Sep 18 18:54:34.917131 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.917099    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-hzt8x"]
Sep 18 18:54:34.920802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:34.920780    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-hzt8x"]
Sep 18 18:54:35.894522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:35.894490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5558864f74-hkh79" event={"ID":"796fe68c-8a7a-4942-82c0-24f6b5277345","Type":"ContainerStarted","Data":"170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839"}
Sep 18 18:54:35.909842 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:35.909793    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5558864f74-hkh79" podStartSLOduration=2.273649031 podStartE2EDuration="2.909778096s" totalImagesPullingTime="636.129065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:54:35.908838961 +0000 UTC m=+599.701515891" watchObservedRunningTime="2026-09-18 18:54:35.909778096 +0000 UTC m=+599.702455025"
Sep 18 18:54:35.935029 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:35.935000    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2s7rs"]
Sep 18 18:54:35.935211 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:35.935186    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2s7rs" podUID="fb4be7cd-2811-4abd-ae21-fcb7baf2b153" containerName="authorino" containerID="cri-o://2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4" gracePeriod=30
Sep 18 18:54:36.166636 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.166616    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2s7rs"
Sep 18 18:54:36.324066 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.324040    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch4fb\" (UniqueName: \"kubernetes.io/projected/fb4be7cd-2811-4abd-ae21-fcb7baf2b153-kube-api-access-ch4fb\") pod \"fb4be7cd-2811-4abd-ae21-fcb7baf2b153\" (UID: \"fb4be7cd-2811-4abd-ae21-fcb7baf2b153\") "
Sep 18 18:54:36.326242 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.326210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4be7cd-2811-4abd-ae21-fcb7baf2b153-kube-api-access-ch4fb" (OuterVolumeSpecName: "kube-api-access-ch4fb") pod "fb4be7cd-2811-4abd-ae21-fcb7baf2b153" (UID: "fb4be7cd-2811-4abd-ae21-fcb7baf2b153"). InnerVolumeSpecName "kube-api-access-ch4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:54:36.335456 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.335433    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53cc878c-3ef5-4774-9312-61602ce4816c" path="/var/lib/kubelet/pods/53cc878c-3ef5-4774-9312-61602ce4816c/volumes"
Sep 18 18:54:36.424666 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.424616    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch4fb\" (UniqueName: \"kubernetes.io/projected/fb4be7cd-2811-4abd-ae21-fcb7baf2b153-kube-api-access-ch4fb\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:54:36.899974 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.899944    2582 generic.go:356] "Generic (PLEG): container finished" podID="fb4be7cd-2811-4abd-ae21-fcb7baf2b153" containerID="2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4" exitCode=0
Sep 18 18:54:36.900259 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.899983    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2s7rs"
Sep 18 18:54:36.900259 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.900035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2s7rs" event={"ID":"fb4be7cd-2811-4abd-ae21-fcb7baf2b153","Type":"ContainerDied","Data":"2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4"}
Sep 18 18:54:36.900259 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.900068    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2s7rs" event={"ID":"fb4be7cd-2811-4abd-ae21-fcb7baf2b153","Type":"ContainerDied","Data":"c75b9cbe1b48b3e19db3c963a3a1cc3e86d4920a04d60a7324d2e4fbede8550d"}
Sep 18 18:54:36.900259 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.900092    2582 scope.go:117] "RemoveContainer" containerID="2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4"
Sep 18 18:54:36.908208 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.908190    2582 scope.go:117] "RemoveContainer" containerID="2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4"
Sep 18 18:54:36.908470 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:54:36.908453    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4\": container with ID starting with 2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4 not found: ID does not exist" containerID="2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4"
Sep 18 18:54:36.908530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.908477    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4"} err="failed to get container status \"2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4\": rpc error: code = NotFound desc = could not find container \"2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4\": container with ID starting with 2dac33a82a02b1bbf424a68271ed3577bd5237ed1e5d5a66ba9abe528f15e6b4 not found: ID does not exist"
Sep 18 18:54:36.915574 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.915554    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2s7rs"]
Sep 18 18:54:36.921153 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:36.921134    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2s7rs"]
Sep 18 18:54:38.334503 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:38.334468    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4be7cd-2811-4abd-ae21-fcb7baf2b153" path="/var/lib/kubelet/pods/fb4be7cd-2811-4abd-ae21-fcb7baf2b153/volumes"
Sep 18 18:54:40.788092 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:40.788058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:40.880630 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:40.880595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:40.984105 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:40.984075    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:41.149900 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:41.149860    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:41.248579 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:41.248553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:44.097212 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:44.097182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:44.203196 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:44.203171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:54:58.280873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:54:58.280781    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:15.641873 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:15.641840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:16.745955 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:16.745929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:17.187145 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:17.187118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:17.272743 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:17.272708    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:17.356445 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:17.356421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:17.604357 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:17.604289    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:17.707653 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:17.707605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:17.796674 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:17.796648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:17.893081 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:17.893047    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:55:23.316951 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:23.316922    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:28.336077 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:28.336046    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:28.403844 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:28.403818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:33.558935 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:33.558878    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:33.613830 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:33.613802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:36.592247 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:36.592220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:39.418233 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:39.418206    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:39.481684 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:39.481656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:51.226691 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:51.226656    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:51.711109 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:51.711067    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:55:58.122825 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:58.122793    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:55:58.821747 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:55:58.821715    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:56:14.801739 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:14.801706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:16.786001 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:16.785964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:17.796708 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:17.796671    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:56:22.169060 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:22.168993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:23.586766 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:23.586727    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:56:25.192360 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:25.192327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:25.253082 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:25.253054    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:31.432035 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:31.432001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:32.982112 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:32.982075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:56:38.285107 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.285075    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"]
Sep 18 18:56:38.285460 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.285395    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4be7cd-2811-4abd-ae21-fcb7baf2b153" containerName="authorino"
Sep 18 18:56:38.285460 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.285406    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4be7cd-2811-4abd-ae21-fcb7baf2b153" containerName="authorino"
Sep 18 18:56:38.285530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.285467    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4be7cd-2811-4abd-ae21-fcb7baf2b153" containerName="authorino"
Sep 18 18:56:38.288340 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.288323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.291179 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.291155    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 18:56:38.291179 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.291181    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 18:56:38.291361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.291155    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 18 18:56:38.291361 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.291158    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lfvwt\""
Sep 18 18:56:38.299257 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.299231    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"]
Sep 18 18:56:38.305083 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.305060    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:38.397614 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.397594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.397720 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.397628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.397720 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.397645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.397720 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.397701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5b8\" (UniqueName: \"kubernetes.io/projected/d9e9860e-1121-447f-9f03-8a34a6430ea4-kube-api-access-hx5b8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.397825 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.397727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.397825 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.397783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9860e-1121-447f-9f03-8a34a6430ea4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499086 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5b8\" (UniqueName: \"kubernetes.io/projected/d9e9860e-1121-447f-9f03-8a34a6430ea4-kube-api-access-hx5b8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9860e-1121-447f-9f03-8a34a6430ea4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499200 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499519 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499581 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.499581 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.499574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.501193 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.501176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e9860e-1121-447f-9f03-8a34a6430ea4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.501536 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.501515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e9860e-1121-447f-9f03-8a34a6430ea4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.507160 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.507137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5b8\" (UniqueName: \"kubernetes.io/projected/d9e9860e-1121-447f-9f03-8a34a6430ea4-kube-api-access-hx5b8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vvwk\" (UID: \"d9e9860e-1121-447f-9f03-8a34a6430ea4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.599176 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.599122    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:38.866738 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.866712    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"]
Sep 18 18:56:38.869102 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:56:38.869076    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e9860e_1121_447f_9f03_8a34a6430ea4.slice/crio-76367b003b02ebd4984e7d37a056a076a0a01cad4444a878f297839f5f82705e WatchSource:0}: Error finding container 76367b003b02ebd4984e7d37a056a076a0a01cad4444a878f297839f5f82705e: Status 404 returned error can't find the container with id 76367b003b02ebd4984e7d37a056a076a0a01cad4444a878f297839f5f82705e
Sep 18 18:56:38.870343 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:38.870321    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 18:56:39.427224 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:39.427181    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk" event={"ID":"d9e9860e-1121-447f-9f03-8a34a6430ea4","Type":"ContainerStarted","Data":"76367b003b02ebd4984e7d37a056a076a0a01cad4444a878f297839f5f82705e"}
Sep 18 18:56:40.284280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:40.284247    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:56:40.434617 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:40.434568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk" event={"ID":"d9e9860e-1121-447f-9f03-8a34a6430ea4","Type":"ContainerStarted","Data":"cd8ae4793e1e2f5c6baadc8bdef5380005193eeafa54d87759880117269d178d"}
Sep 18 18:56:40.457374 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:40.457328    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk" podStartSLOduration=1.4659115059999999 podStartE2EDuration="2.457312316s" totalImagesPullingTime="991.40081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:56:40.45655959 +0000 UTC m=+724.249236510" watchObservedRunningTime="2026-09-18 18:56:40.457312316 +0000 UTC m=+724.249989280"
Sep 18 18:56:47.207075 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.207041    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"]
Sep 18 18:56:47.210391 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.210374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.212665 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.212643    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 18:56:47.224847 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.224827    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"]
Sep 18 18:56:47.265265 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.265238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.265371 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.265284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d120fce4-2afd-49c2-b6d0-50fe4fee252a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.265371 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.265340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlk5\" (UniqueName: \"kubernetes.io/projected/d120fce4-2afd-49c2-b6d0-50fe4fee252a-kube-api-access-hrlk5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.265458 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.265388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.265458 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.265445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.265538 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.265480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.291393 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.291372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:56:47.365936 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.365910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlk5\" (UniqueName: \"kubernetes.io/projected/d120fce4-2afd-49c2-b6d0-50fe4fee252a-kube-api-access-hrlk5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.365946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.365986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366021 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.366008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366157 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.366107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366157 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.366137    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d120fce4-2afd-49c2-b6d0-50fe4fee252a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366440 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.366407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366535 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.366436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.366535 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.366458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.368198 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.368178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d120fce4-2afd-49c2-b6d0-50fe4fee252a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.368511 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.368492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d120fce4-2afd-49c2-b6d0-50fe4fee252a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.374812 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.374793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlk5\" (UniqueName: \"kubernetes.io/projected/d120fce4-2afd-49c2-b6d0-50fe4fee252a-kube-api-access-hrlk5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-knpz4\" (UID: \"d120fce4-2afd-49c2-b6d0-50fe4fee252a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.518445 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.518394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:47.787643 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:47.787609    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"]
Sep 18 18:56:47.788836 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:56:47.788811    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd120fce4_2afd_49c2_b6d0_50fe4fee252a.slice/crio-3605a546c07b806a5b63ddec3a597324be458dc0f32365358778319e34096ae0 WatchSource:0}: Error finding container 3605a546c07b806a5b63ddec3a597324be458dc0f32365358778319e34096ae0: Status 404 returned error can't find the container with id 3605a546c07b806a5b63ddec3a597324be458dc0f32365358778319e34096ae0
Sep 18 18:56:48.469518 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:48.469484    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4" event={"ID":"d120fce4-2afd-49c2-b6d0-50fe4fee252a","Type":"ContainerStarted","Data":"1d321d2f9d0f65b99247af122601546da6e4cd1618995dedd702405e9cb71f49"}
Sep 18 18:56:48.469829 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:48.469530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4" event={"ID":"d120fce4-2afd-49c2-b6d0-50fe4fee252a","Type":"ContainerStarted","Data":"3605a546c07b806a5b63ddec3a597324be458dc0f32365358778319e34096ae0"}
Sep 18 18:56:48.486706 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:48.486664    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4" podStartSLOduration=1.3049466920000001 podStartE2EDuration="1.48664944s" totalImagesPullingTime="181.702748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:56:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:56:48.485107214 +0000 UTC m=+732.277784144" watchObservedRunningTime="2026-09-18 18:56:48.48664944 +0000 UTC m=+732.279326349"
Sep 18 18:56:48.599559 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:48.599536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:48.599665 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:48.599605    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:48.611240 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:48.611219    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:49.485255 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:49.485235    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vvwk"
Sep 18 18:56:49.693322 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:49.693288    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:56:57.519522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:57.519493    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:57.519522 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:57.519525    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:57.531102 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:57.531072    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:56:58.526099 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:56:58.526068    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-knpz4"
Sep 18 18:58:25.528050 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.527976    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd68ff6d7-mjb6f"]
Sep 18 18:58:25.530101 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.530085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:25.541922 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.541877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd68ff6d7-mjb6f"]
Sep 18 18:58:25.581471 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.581446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7gj\" (UniqueName: \"kubernetes.io/projected/7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de-kube-api-access-8t7gj\") pod \"authorino-6dd68ff6d7-mjb6f\" (UID: \"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de\") " pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:25.581578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.581515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de-tls-cert\") pod \"authorino-6dd68ff6d7-mjb6f\" (UID: \"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de\") " pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:25.681978 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.681955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7gj\" (UniqueName: \"kubernetes.io/projected/7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de-kube-api-access-8t7gj\") pod \"authorino-6dd68ff6d7-mjb6f\" (UID: \"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de\") " pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:25.682099 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.682014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de-tls-cert\") pod \"authorino-6dd68ff6d7-mjb6f\" (UID: \"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de\") " pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:25.684198 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.684174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de-tls-cert\") pod \"authorino-6dd68ff6d7-mjb6f\" (UID: \"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de\") " pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:25.691328 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.691306    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7gj\" (UniqueName: \"kubernetes.io/projected/7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de-kube-api-access-8t7gj\") pod \"authorino-6dd68ff6d7-mjb6f\" (UID: \"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de\") " pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:25.838119 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:25.838064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f"
Sep 18 18:58:26.107995 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:26.107970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd68ff6d7-mjb6f"]
Sep 18 18:58:26.109910 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:58:26.109867    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f0cd22e_f4c3_4f6c_83cf_547e2ac5e5de.slice/crio-fd3b8151004ddefabcc69b504846005366da4a9803ed1124ba8ff26db30f2e34 WatchSource:0}: Error finding container fd3b8151004ddefabcc69b504846005366da4a9803ed1124ba8ff26db30f2e34: Status 404 returned error can't find the container with id fd3b8151004ddefabcc69b504846005366da4a9803ed1124ba8ff26db30f2e34
Sep 18 18:58:26.892852 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:26.892821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f" event={"ID":"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de","Type":"ContainerStarted","Data":"fd3b8151004ddefabcc69b504846005366da4a9803ed1124ba8ff26db30f2e34"}
Sep 18 18:58:28.904930 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:28.904873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f" event={"ID":"7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de","Type":"ContainerStarted","Data":"7cad1a102d3793b133b626ba5bfc5b4273f305ef2755a4d16506a98ef1225460"}
Sep 18 18:58:28.942542 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:28.942485    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6dd68ff6d7-mjb6f" podStartSLOduration=1.756747781 podStartE2EDuration="3.942470585s" totalImagesPullingTime="2.185722804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:58:28.941582122 +0000 UTC m=+832.734259053" watchObservedRunningTime="2026-09-18 18:58:28.942470585 +0000 UTC m=+832.735147515"
Sep 18 18:58:28.976452 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:28.976418    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5558864f74-hkh79"]
Sep 18 18:58:28.976667 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:28.976640    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5558864f74-hkh79" podUID="796fe68c-8a7a-4942-82c0-24f6b5277345" containerName="authorino" containerID="cri-o://170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839" gracePeriod=30
Sep 18 18:58:29.217940 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.217909    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:58:29.305901 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.305842    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/796fe68c-8a7a-4942-82c0-24f6b5277345-tls-cert\") pod \"796fe68c-8a7a-4942-82c0-24f6b5277345\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") "
Sep 18 18:58:29.306073 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.306021    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrsb7\" (UniqueName: \"kubernetes.io/projected/796fe68c-8a7a-4942-82c0-24f6b5277345-kube-api-access-wrsb7\") pod \"796fe68c-8a7a-4942-82c0-24f6b5277345\" (UID: \"796fe68c-8a7a-4942-82c0-24f6b5277345\") "
Sep 18 18:58:29.308155 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.308122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796fe68c-8a7a-4942-82c0-24f6b5277345-kube-api-access-wrsb7" (OuterVolumeSpecName: "kube-api-access-wrsb7") pod "796fe68c-8a7a-4942-82c0-24f6b5277345" (UID: "796fe68c-8a7a-4942-82c0-24f6b5277345"). InnerVolumeSpecName "kube-api-access-wrsb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:29.322454 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.322155    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796fe68c-8a7a-4942-82c0-24f6b5277345-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "796fe68c-8a7a-4942-82c0-24f6b5277345" (UID: "796fe68c-8a7a-4942-82c0-24f6b5277345"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:58:29.407712 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.407675    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrsb7\" (UniqueName: \"kubernetes.io/projected/796fe68c-8a7a-4942-82c0-24f6b5277345-kube-api-access-wrsb7\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:58:29.407712 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.407706    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/796fe68c-8a7a-4942-82c0-24f6b5277345-tls-cert\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:58:29.716853 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.716819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:29.876976 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.876929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:29.911643 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.911608    2582 generic.go:356] "Generic (PLEG): container finished" podID="796fe68c-8a7a-4942-82c0-24f6b5277345" containerID="170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839" exitCode=0
Sep 18 18:58:29.912110 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.911656    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5558864f74-hkh79"
Sep 18 18:58:29.912110 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.911695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5558864f74-hkh79" event={"ID":"796fe68c-8a7a-4942-82c0-24f6b5277345","Type":"ContainerDied","Data":"170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839"}
Sep 18 18:58:29.912110 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.911745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5558864f74-hkh79" event={"ID":"796fe68c-8a7a-4942-82c0-24f6b5277345","Type":"ContainerDied","Data":"25fcbcaedfff0c2b97def42b72e908ed49e843cba84b6000714ccbcd1a4d3227"}
Sep 18 18:58:29.912110 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.911764    2582 scope.go:117] "RemoveContainer" containerID="170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839"
Sep 18 18:58:29.921502 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.921484    2582 scope.go:117] "RemoveContainer" containerID="170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839"
Sep 18 18:58:29.921801 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:58:29.921782    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839\": container with ID starting with 170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839 not found: ID does not exist" containerID="170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839"
Sep 18 18:58:29.921911 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.921816    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839"} err="failed to get container status \"170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839\": rpc error: code = NotFound desc = could not find container \"170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839\": container with ID starting with 170f98d84af458c721ecdc898c282fb4bbb0fb16c3a017cef514ef033835a839 not found: ID does not exist"
Sep 18 18:58:29.973589 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.973519    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5558864f74-hkh79"]
Sep 18 18:58:29.985301 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:29.985265    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5558864f74-hkh79"]
Sep 18 18:58:30.334646 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:30.334569    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796fe68c-8a7a-4942-82c0-24f6b5277345" path="/var/lib/kubelet/pods/796fe68c-8a7a-4942-82c0-24f6b5277345/volumes"
Sep 18 18:58:30.936027 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:30.935997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:31.127302 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:31.127276    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:46.894936 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:46.894877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:46.996003 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:46.995974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:47.129719 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:47.129695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:47.385207 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:47.385131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:47.616604 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:47.616578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:47.764134 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:47.764110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:47.885403 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:47.885378    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:47.989913 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:47.989872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:48.526025 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:48.525999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:48.734964 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:48.734932    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:48.991759 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:48.991740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:49.114231 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.114205    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-76b973"]
Sep 18 18:58:49.114510 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.114499    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796fe68c-8a7a-4942-82c0-24f6b5277345" containerName="authorino"
Sep 18 18:58:49.114565 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.114512    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="796fe68c-8a7a-4942-82c0-24f6b5277345" containerName="authorino"
Sep 18 18:58:49.114607 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.114582    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="796fe68c-8a7a-4942-82c0-24f6b5277345" containerName="authorino"
Sep 18 18:58:49.118833 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.118819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:49.121800 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.121775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wgm9b\""
Sep 18 18:58:49.153756 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.153735    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-76b973"]
Sep 18 18:58:49.250268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.250220    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s976\" (UniqueName: \"kubernetes.io/projected/5c7f0db1-ca83-4249-8b9d-358489f48c98-kube-api-access-2s976\") pod \"test-curl-7693-76b973\" (UID: \"5c7f0db1-ca83-4249-8b9d-358489f48c98\") " pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:49.350803 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.350780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s976\" (UniqueName: \"kubernetes.io/projected/5c7f0db1-ca83-4249-8b9d-358489f48c98-kube-api-access-2s976\") pod \"test-curl-7693-76b973\" (UID: \"5c7f0db1-ca83-4249-8b9d-358489f48c98\") " pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:49.361252 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.361233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s976\" (UniqueName: \"kubernetes.io/projected/5c7f0db1-ca83-4249-8b9d-358489f48c98-kube-api-access-2s976\") pod \"test-curl-7693-76b973\" (UID: \"5c7f0db1-ca83-4249-8b9d-358489f48c98\") " pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:49.426751 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.426731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:49.708644 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:49.708614    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-76b973"]
Sep 18 18:58:49.711642 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:58:49.711612    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c7f0db1_ca83_4249_8b9d_358489f48c98.slice/crio-54c3a4b2e2aebe9239458d49c3c4d898f096d22260bb385a6d6e6f32fad4a5fd WatchSource:0}: Error finding container 54c3a4b2e2aebe9239458d49c3c4d898f096d22260bb385a6d6e6f32fad4a5fd: Status 404 returned error can't find the container with id 54c3a4b2e2aebe9239458d49c3c4d898f096d22260bb385a6d6e6f32fad4a5fd
Sep 18 18:58:50.010042 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:50.009983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-76b973" event={"ID":"5c7f0db1-ca83-4249-8b9d-358489f48c98","Type":"ContainerStarted","Data":"54c3a4b2e2aebe9239458d49c3c4d898f096d22260bb385a6d6e6f32fad4a5fd"}
Sep 18 18:58:50.636154 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:50.636128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:50.811859 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:50.811833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:51.195529 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:51.195491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:51.283835 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:51.283797    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:58:51.411180 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:51.411129    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:51.641835 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:51.641802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:51.812898 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:51.812856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:53.027977 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:53.027948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-76b973" event={"ID":"5c7f0db1-ca83-4249-8b9d-358489f48c98","Type":"ContainerStarted","Data":"0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6"}
Sep 18 18:58:53.057316 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:53.057264    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-76b973" podStartSLOduration=1.667847093 podStartE2EDuration="4.057246946s" totalImagesPullingTime="2.389399853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:58:53.056492336 +0000 UTC m=+856.849169267" watchObservedRunningTime="2026-09-18 18:58:53.057246946 +0000 UTC m=+856.849923876"
Sep 18 18:58:53.078913 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:53.078849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:53.222547 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:53.222526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:53.486959 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:53.486939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:54.057491 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.057464    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-76b973"]
Sep 18 18:58:54.072985 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.072965    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-76b973"]
Sep 18 18:58:54.237843 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.237812    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-993d33"]
Sep 18 18:58:54.238192 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.238174    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7f0db1-ca83-4249-8b9d-358489f48c98" containerName="test-curl-7693-76b973"
Sep 18 18:58:54.238282 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.238222    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7f0db1-ca83-4249-8b9d-358489f48c98" containerName="test-curl-7693-76b973"
Sep 18 18:58:54.238357 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.238345    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7f0db1-ca83-4249-8b9d-358489f48c98" containerName="test-curl-7693-76b973"
Sep 18 18:58:54.241136 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.241118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:58:54.268216 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.268189    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-993d33"]
Sep 18 18:58:54.391026 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.390956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nlxv\" (UniqueName: \"kubernetes.io/projected/b896169d-a56b-488a-8789-386809a2e97a-kube-api-access-2nlxv\") pod \"test-curl-7693-993d33\" (UID: \"b896169d-a56b-488a-8789-386809a2e97a\") " pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:58:54.454922 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.454875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:54.492144 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.492119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nlxv\" (UniqueName: \"kubernetes.io/projected/b896169d-a56b-488a-8789-386809a2e97a-kube-api-access-2nlxv\") pod \"test-curl-7693-993d33\" (UID: \"b896169d-a56b-488a-8789-386809a2e97a\") " pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:58:54.531415 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.531387    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nlxv\" (UniqueName: \"kubernetes.io/projected/b896169d-a56b-488a-8789-386809a2e97a-kube-api-access-2nlxv\") pod \"test-curl-7693-993d33\" (UID: \"b896169d-a56b-488a-8789-386809a2e97a\") " pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:58:54.549314 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.549286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:58:54.841386 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.841318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-993d33"]
Sep 18 18:58:54.841606 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:58:54.841575    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb896169d_a56b_488a_8789_386809a2e97a.slice/crio-1d97a32d71a86ebde1c643ade5f2b759c107423c240e375e978c9d1201723192 WatchSource:0}: Error finding container 1d97a32d71a86ebde1c643ade5f2b759c107423c240e375e978c9d1201723192: Status 404 returned error can't find the container with id 1d97a32d71a86ebde1c643ade5f2b759c107423c240e375e978c9d1201723192
Sep 18 18:58:54.862861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:54.862840    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:55.038961 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:55.038928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-993d33" event={"ID":"b896169d-a56b-488a-8789-386809a2e97a","Type":"ContainerStarted","Data":"f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee"}
Sep 18 18:58:55.039077 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:55.038970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-993d33" event={"ID":"b896169d-a56b-488a-8789-386809a2e97a","Type":"ContainerStarted","Data":"1d97a32d71a86ebde1c643ade5f2b759c107423c240e375e978c9d1201723192"}
Sep 18 18:58:55.039144 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:55.039124    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-76b973" podUID="5c7f0db1-ca83-4249-8b9d-358489f48c98" containerName="test-curl-7693-76b973" containerID="cri-o://0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6" gracePeriod=2
Sep 18 18:58:55.067232 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:55.067191    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-993d33" podStartSLOduration=1.067177949 podStartE2EDuration="1.067177949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:58:55.065205808 +0000 UTC m=+858.857882729" watchObservedRunningTime="2026-09-18 18:58:55.067177949 +0000 UTC m=+858.859854879"
Sep 18 18:58:55.502072 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:55.502043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:55.753340 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:55.753311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:56.142488 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.142463    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-993d33"]
Sep 18 18:58:56.151507 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.151478    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-993d33"]
Sep 18 18:58:56.581308 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.581230    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:56.784765 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.784720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:56.883962 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.883894    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"]
Sep 18 18:58:56.884216 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.884205    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b896169d-a56b-488a-8789-386809a2e97a" containerName="test-curl-7693-993d33"
Sep 18 18:58:56.884258 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.884218    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b896169d-a56b-488a-8789-386809a2e97a" containerName="test-curl-7693-993d33"
Sep 18 18:58:56.884301 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.884293    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b896169d-a56b-488a-8789-386809a2e97a" containerName="test-curl-7693-993d33"
Sep 18 18:58:56.887280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.887266    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:56.891793 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.891770    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-942883-dockercfg-vfs9q\""
Sep 18 18:58:56.892236 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.892213    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-942883\""
Sep 18 18:58:56.928745 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.928726    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"]
Sep 18 18:58:56.980705 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:56.980687    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:57.011740 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.011721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e1962-ebd5-4d7b-9ea1-b7a055f69977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:57.011823 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.011755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkwj\" (UniqueName: \"kubernetes.io/projected/804e1962-ebd5-4d7b-9ea1-b7a055f69977-kube-api-access-xjkwj\") pod \"payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:57.048141 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.048113    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-993d33" podUID="b896169d-a56b-488a-8789-386809a2e97a" containerName="test-curl-7693-993d33" containerID="cri-o://f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee" gracePeriod=2
Sep 18 18:58:57.112355 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.112311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e1962-ebd5-4d7b-9ea1-b7a055f69977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:57.112493 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.112370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkwj\" (UniqueName: \"kubernetes.io/projected/804e1962-ebd5-4d7b-9ea1-b7a055f69977-kube-api-access-xjkwj\") pod \"payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:57.113537 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.113439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e1962-ebd5-4d7b-9ea1-b7a055f69977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:57.128384 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.128355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkwj\" (UniqueName: \"kubernetes.io/projected/804e1962-ebd5-4d7b-9ea1-b7a055f69977-kube-api-access-xjkwj\") pod \"payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:57.180487 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.180468    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:57.197490 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.197469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:57.314621 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.314592    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s976\" (UniqueName: \"kubernetes.io/projected/5c7f0db1-ca83-4249-8b9d-358489f48c98-kube-api-access-2s976\") pod \"5c7f0db1-ca83-4249-8b9d-358489f48c98\" (UID: \"5c7f0db1-ca83-4249-8b9d-358489f48c98\") "
Sep 18 18:58:57.317040 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.317014    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7f0db1-ca83-4249-8b9d-358489f48c98-kube-api-access-2s976" (OuterVolumeSpecName: "kube-api-access-2s976") pod "5c7f0db1-ca83-4249-8b9d-358489f48c98" (UID: "5c7f0db1-ca83-4249-8b9d-358489f48c98"). InnerVolumeSpecName "kube-api-access-2s976". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:57.397928 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.397876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:57.415768 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.415721    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s976\" (UniqueName: \"kubernetes.io/projected/5c7f0db1-ca83-4249-8b9d-358489f48c98-kube-api-access-2s976\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:58:57.535235 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:58:57.535214    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804e1962_ebd5_4d7b_9ea1_b7a055f69977.slice/crio-4e80e681110d092881296c65b7d1b49cb53898f131a3456b5a158c6df3c83636 WatchSource:0}: Error finding container 4e80e681110d092881296c65b7d1b49cb53898f131a3456b5a158c6df3c83636: Status 404 returned error can't find the container with id 4e80e681110d092881296c65b7d1b49cb53898f131a3456b5a158c6df3c83636
Sep 18 18:58:57.544896 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.544860    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"]
Sep 18 18:58:57.789985 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.789925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:57.819876 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.819850    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"]
Sep 18 18:58:57.824315 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.824298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:57.828070 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.828052    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 18:58:57.828183 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.828119    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 18:58:57.829983 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.829965    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 18:58:57.830334 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.830319    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1a444c-serving-cert\""
Sep 18 18:58:57.830521 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.830508    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f6fxk\""
Sep 18 18:58:57.865173 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.865153    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"]
Sep 18 18:58:57.920096 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.920069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:57.920204 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.920135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:57.920204 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.920165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4zm\" (UniqueName: \"kubernetes.io/projected/74283389-8b96-40fd-b345-a91801ec8261-kube-api-access-sf4zm\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:57.920204 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:57.920192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74283389-8b96-40fd-b345-a91801ec8261-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.021351 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.021327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.021457 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.021379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.021457 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.021400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4zm\" (UniqueName: \"kubernetes.io/projected/74283389-8b96-40fd-b345-a91801ec8261-kube-api-access-sf4zm\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.021457 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.021428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74283389-8b96-40fd-b345-a91801ec8261-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.022106 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.022088    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74283389-8b96-40fd-b345-a91801ec8261-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.023847 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.023823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.023847 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.023831    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.039265 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.039245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4zm\" (UniqueName: \"kubernetes.io/projected/74283389-8b96-40fd-b345-a91801ec8261-kube-api-access-sf4zm\") pod \"maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.054232 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.054033    2582 generic.go:356] "Generic (PLEG): container finished" podID="5c7f0db1-ca83-4249-8b9d-358489f48c98" containerID="0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6" exitCode=137
Sep 18 18:58:58.054232 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.054090    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:58.054232 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.054134    2582 scope.go:117] "RemoveContainer" containerID="0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6"
Sep 18 18:58:58.057001 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.056231    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr" event={"ID":"804e1962-ebd5-4d7b-9ea1-b7a055f69977","Type":"ContainerStarted","Data":"4e80e681110d092881296c65b7d1b49cb53898f131a3456b5a158c6df3c83636"}
Sep 18 18:58:58.064987 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.064967    2582 scope.go:117] "RemoveContainer" containerID="0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6"
Sep 18 18:58:58.065331 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:58:58.065281    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6\": container with ID starting with 0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6 not found: ID does not exist" containerID="0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6"
Sep 18 18:58:58.065331 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.065315    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6"} err="failed to get container status \"0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6\": rpc error: code = NotFound desc = could not find container \"0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6\": container with ID starting with 0f2333bdeaa5eb7988dbffdd537461c3106a99e3462495ddd9af37f65ec04aa6 not found: ID does not exist"
Sep 18 18:58:58.065507 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:58:58.065478    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-76b973\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5c7f0db1-ca83-4249-8b9d-358489f48c98" pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:58.076752 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.076729    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:58.134824 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.134799    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:58:58.335002 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:58:58.334931    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-76b973\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="5c7f0db1-ca83-4249-8b9d-358489f48c98" pod="opendatahub/test-curl-7693-76b973"
Sep 18 18:58:58.337354 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.337324    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7f0db1-ca83-4249-8b9d-358489f48c98" path="/var/lib/kubelet/pods/5c7f0db1-ca83-4249-8b9d-358489f48c98/volumes"
Sep 18 18:58:58.458590 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.458567    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"]
Sep 18 18:58:58.670535 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:58:58.670507    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74283389_8b96_40fd_b345_a91801ec8261.slice/crio-ea04f5c22f3ce0767f1c572b854757c64127fb51622c6e3caf1138ce6939a8a1 WatchSource:0}: Error finding container ea04f5c22f3ce0767f1c572b854757c64127fb51622c6e3caf1138ce6939a8a1: Status 404 returned error can't find the container with id ea04f5c22f3ce0767f1c572b854757c64127fb51622c6e3caf1138ce6939a8a1
Sep 18 18:58:58.863234 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:58.863207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:59.044136 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.044056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:59.063627 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.063598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt" event={"ID":"74283389-8b96-40fd-b345-a91801ec8261","Type":"ContainerStarted","Data":"ea04f5c22f3ce0767f1c572b854757c64127fb51622c6e3caf1138ce6939a8a1"}
Sep 18 18:58:59.065375 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.065352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr" event={"ID":"804e1962-ebd5-4d7b-9ea1-b7a055f69977","Type":"ContainerStarted","Data":"2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4"}
Sep 18 18:58:59.065470 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.065443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:58:59.115492 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.115429    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr" podStartSLOduration=1.936598338 podStartE2EDuration="3.115409525s" totalImagesPullingTime="1.178811187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:58:59.111227997 +0000 UTC m=+862.903904927" watchObservedRunningTime="2026-09-18 18:58:59.115409525 +0000 UTC m=+862.908086463"
Sep 18 18:58:59.191708 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.191679    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"]
Sep 18 18:58:59.195316 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.195298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.199253 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.199232    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1a444c-dockercfg-jrf8b\""
Sep 18 18:58:59.200318 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.200298    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1a444c\""
Sep 18 18:58:59.249688 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.249647    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"]
Sep 18 18:58:59.258655 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.258631    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:58:59.335159 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.335046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zffn\" (UniqueName: \"kubernetes.io/projected/32ebcb1a-c7a9-4c94-a87e-2453a042765e-kube-api-access-6zffn\") pod \"payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.335597 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.335157    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ebcb1a-c7a9-4c94-a87e-2453a042765e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.436167 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.436127    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nlxv\" (UniqueName: \"kubernetes.io/projected/b896169d-a56b-488a-8789-386809a2e97a-kube-api-access-2nlxv\") pod \"b896169d-a56b-488a-8789-386809a2e97a\" (UID: \"b896169d-a56b-488a-8789-386809a2e97a\") "
Sep 18 18:58:59.436406 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.436381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ebcb1a-c7a9-4c94-a87e-2453a042765e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.436534 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.436490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zffn\" (UniqueName: \"kubernetes.io/projected/32ebcb1a-c7a9-4c94-a87e-2453a042765e-kube-api-access-6zffn\") pod \"payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.437797 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.437733    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ebcb1a-c7a9-4c94-a87e-2453a042765e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.440767 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.440736    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b896169d-a56b-488a-8789-386809a2e97a-kube-api-access-2nlxv" (OuterVolumeSpecName: "kube-api-access-2nlxv") pod "b896169d-a56b-488a-8789-386809a2e97a" (UID: "b896169d-a56b-488a-8789-386809a2e97a"). InnerVolumeSpecName "kube-api-access-2nlxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:58:59.447157 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.447116    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zffn\" (UniqueName: \"kubernetes.io/projected/32ebcb1a-c7a9-4c94-a87e-2453a042765e-kube-api-access-6zffn\") pod \"payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.508396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.508370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:58:59.537474 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.537435    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nlxv\" (UniqueName: \"kubernetes.io/projected/b896169d-a56b-488a-8789-386809a2e97a-kube-api-access-2nlxv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:58:59.694802 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.694764    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:58:59.841120 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:58:59.841022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"]
Sep 18 18:58:59.841526 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:58:59.841488    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ebcb1a_c7a9_4c94_a87e_2453a042765e.slice/crio-f24bb4062d4fd201cc913a53ae84b12a5c278516bf889151e7ed0a807d31edf0 WatchSource:0}: Error finding container f24bb4062d4fd201cc913a53ae84b12a5c278516bf889151e7ed0a807d31edf0: Status 404 returned error can't find the container with id f24bb4062d4fd201cc913a53ae84b12a5c278516bf889151e7ed0a807d31edf0
Sep 18 18:59:00.073245 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.073212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn" event={"ID":"32ebcb1a-c7a9-4c94-a87e-2453a042765e","Type":"ContainerStarted","Data":"c95cf18d77ab809ec0ae78a7a61c7cb84b1800884304b60fbad672569e022e71"}
Sep 18 18:59:00.073413 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.073385    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:59:00.073479 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.073428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn" event={"ID":"32ebcb1a-c7a9-4c94-a87e-2453a042765e","Type":"ContainerStarted","Data":"f24bb4062d4fd201cc913a53ae84b12a5c278516bf889151e7ed0a807d31edf0"}
Sep 18 18:59:00.074602 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.074573    2582 generic.go:356] "Generic (PLEG): container finished" podID="b896169d-a56b-488a-8789-386809a2e97a" containerID="f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee" exitCode=137
Sep 18 18:59:00.074800 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.074784    2582 scope.go:117] "RemoveContainer" containerID="f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee"
Sep 18 18:59:00.074964 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.074946    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:59:00.163103 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.163040    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn" podStartSLOduration=1.163018906 podStartE2EDuration="1.163018906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:00.161357371 +0000 UTC m=+863.954034328" watchObservedRunningTime="2026-09-18 18:59:00.163018906 +0000 UTC m=+863.955695837"
Sep 18 18:59:00.198133 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:59:00.198089    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7693-993d33\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="b896169d-a56b-488a-8789-386809a2e97a" pod="opendatahub/test-curl-7693-993d33"
Sep 18 18:59:00.335563 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.335532    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b896169d-a56b-488a-8789-386809a2e97a" path="/var/lib/kubelet/pods/b896169d-a56b-488a-8789-386809a2e97a/volumes"
Sep 18 18:59:00.848729 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.848693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:00.997364 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:00.997330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:01.184428 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:01.184206    2582 scope.go:117] "RemoveContainer" containerID="f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee"
Sep 18 18:59:01.184558 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:59:01.184541    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee\": container with ID starting with f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee not found: ID does not exist" containerID="f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee"
Sep 18 18:59:01.184606 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:01.184570    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee"} err="failed to get container status \"f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee\": rpc error: code = NotFound desc = could not find container \"f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee\": container with ID starting with f23f508beab24adc06b7d1e7ff8c9ca25d53091fe2217e90e54bf3afe5db26ee not found: ID does not exist"
Sep 18 18:59:01.787035 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:01.787009    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:02.088530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:02.088465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt" event={"ID":"74283389-8b96-40fd-b345-a91801ec8261","Type":"ContainerStarted","Data":"f110125b39c93a4cdbcff438db4d53892ebdab5da9037abd0591817d91a9aff7"}
Sep 18 18:59:02.088657 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:02.088595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:59:02.188136 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:02.188095    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt" podStartSLOduration=2.626010877 podStartE2EDuration="5.188082648s" totalImagesPullingTime="2.562071771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:02.182199662 +0000 UTC m=+865.974876605" watchObservedRunningTime="2026-09-18 18:59:02.188082648 +0000 UTC m=+865.980759578"
Sep 18 18:59:03.312070 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:03.312045    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:03.432233 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:03.432211    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:03.591396 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:03.591298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:03.753285 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:03.753264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:04.191720 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:04.191691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:04.434134 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:04.434087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:05.755757 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:05.755726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:05.928972 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:05.928944    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:06.221525 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:06.221498    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:06.465826 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:06.465804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:06.940841 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:06.940809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:07.193419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:07.193346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:07.488043 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:07.487961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:07.849703 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:07.849624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:08.103966 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:08.103911    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 18:59:08.551555 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:08.551526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:08.812259 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:08.812167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:09.045909 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:09.045849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:09.137297 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:09.137213    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:59:09.399010 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:09.398980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:10.077591 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:10.077558    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 18:59:10.770606 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:10.770576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:10.923770 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:10.923745    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:11.083254 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:11.083202    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 18:59:11.701750 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:11.701723    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:11.983204 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:11.983108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:13.760751 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:13.760717    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:14.129684 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:14.129613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:15.392196 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:15.392165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:16.300759 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.300733    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:16.897333 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.897286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:16.958383 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.958351    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"]
Sep 18 18:59:16.961895 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.961868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:16.974235 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.974208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:16.974332 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.974245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-metrics-certs\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:16.974379 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.974333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/085997cb-cd70-4b5c-91de-598c3590e647-kube-api-access-wkqsb\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:16.974415 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.974392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085997cb-cd70-4b5c-91de-598c3590e647-openshift-service-ca\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:16.976750 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:16.976736    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8cbeacf3-serving-cert\""
Sep 18 18:59:17.016376 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.016349    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"]
Sep 18 18:59:17.074933 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.074912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/085997cb-cd70-4b5c-91de-598c3590e647-kube-api-access-wkqsb\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.075037 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.074978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085997cb-cd70-4b5c-91de-598c3590e647-openshift-service-ca\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.075037 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.075015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.075155 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.075035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-metrics-certs\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.075211 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:59:17.075150    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-serving-cert: secret "maas-api-e2e-ait-8cbeacf3-serving-cert" not found
Sep 18 18:59:17.075263 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:59:17.075251    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls podName:085997cb-cd70-4b5c-91de-598c3590e647 nodeName:}" failed. No retries permitted until 2026-09-18 18:59:17.575230463 +0000 UTC m=+881.367907388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls") pod "maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6" (UID: "085997cb-cd70-4b5c-91de-598c3590e647") : secret "maas-api-e2e-ait-8cbeacf3-serving-cert" not found
Sep 18 18:59:17.075590 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.075572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085997cb-cd70-4b5c-91de-598c3590e647-openshift-service-ca\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.077333 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.077314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-metrics-certs\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.111829 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.111805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/085997cb-cd70-4b5c-91de-598c3590e647-kube-api-access-wkqsb\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.220917 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.220836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:17.520813 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.520762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:17.578441 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.578422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.580780 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.580764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls\") pod \"maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.594678 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.594660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:17.899307 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:17.899234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"]
Sep 18 18:59:17.901418 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:17.901391    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085997cb_cd70_4b5c_91de_598c3590e647.slice/crio-25825a9bdddbf2dd5dc180a2c33ab3f67d9af22681ddf62a2104dfafa96627df WatchSource:0}: Error finding container 25825a9bdddbf2dd5dc180a2c33ab3f67d9af22681ddf62a2104dfafa96627df: Status 404 returned error can't find the container with id 25825a9bdddbf2dd5dc180a2c33ab3f67d9af22681ddf62a2104dfafa96627df
Sep 18 18:59:18.170551 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.170497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6" event={"ID":"085997cb-cd70-4b5c-91de-598c3590e647","Type":"ContainerStarted","Data":"25825a9bdddbf2dd5dc180a2c33ab3f67d9af22681ddf62a2104dfafa96627df"}
Sep 18 18:59:18.326180 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.326159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:18.487099 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.487021    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:18.565682 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.565653    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"]
Sep 18 18:59:18.569094 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.569076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.571715 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.571691    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8cbeacf3-dockercfg-876s7\""
Sep 18 18:59:18.572114 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.572092    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8cbeacf3\""
Sep 18 18:59:18.585540 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.585509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae80296c-c554-4668-9e80-2f64e37e81a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.585646 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.585548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8k76\" (UniqueName: \"kubernetes.io/projected/ae80296c-c554-4668-9e80-2f64e37e81a9-kube-api-access-f8k76\") pod \"payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.587579 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.587555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"]
Sep 18 18:59:18.594975 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.594955    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"]
Sep 18 18:59:18.598198 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.598178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:18.619462 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.619439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"]
Sep 18 18:59:18.688655 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.687372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae80296c-c554-4668-9e80-2f64e37e81a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.688655 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.687425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e369bc-6706-468f-b542-0d7e250c165f-plugins-config-volume\") pod \"payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") " pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:18.688655 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.687466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8k76\" (UniqueName: \"kubernetes.io/projected/ae80296c-c554-4668-9e80-2f64e37e81a9-kube-api-access-f8k76\") pod \"payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.688655 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.687508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcn2\" (UniqueName: \"kubernetes.io/projected/f3e369bc-6706-468f-b542-0d7e250c165f-kube-api-access-jwcn2\") pod \"payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") " pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:18.688655 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.688363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae80296c-c554-4668-9e80-2f64e37e81a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.707690 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.707664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8k76\" (UniqueName: \"kubernetes.io/projected/ae80296c-c554-4668-9e80-2f64e37e81a9-kube-api-access-f8k76\") pod \"payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.788189 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.788088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e369bc-6706-468f-b542-0d7e250c165f-plugins-config-volume\") pod \"payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") " pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:18.788189 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.788145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcn2\" (UniqueName: \"kubernetes.io/projected/f3e369bc-6706-468f-b542-0d7e250c165f-kube-api-access-jwcn2\") pod \"payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") " pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:18.789004 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.788982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e369bc-6706-468f-b542-0d7e250c165f-plugins-config-volume\") pod \"payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") " pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:18.807953 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.807932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcn2\" (UniqueName: \"kubernetes.io/projected/f3e369bc-6706-468f-b542-0d7e250c165f-kube-api-access-jwcn2\") pod \"payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") " pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:18.879410 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.879386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:18.907735 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:18.907714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:19.129461 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.129396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:19.177326 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.177294    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"]
Sep 18 18:59:19.178152 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.178126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6" event={"ID":"085997cb-cd70-4b5c-91de-598c3590e647","Type":"ContainerStarted","Data":"7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545"}
Sep 18 18:59:19.178265 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.178249    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:19.179258 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:19.179239    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae80296c_c554_4668_9e80_2f64e37e81a9.slice/crio-01eea6dc54ac13bbfb07632798289c33eb52a4a84c2385eba095dab967b9704d WatchSource:0}: Error finding container 01eea6dc54ac13bbfb07632798289c33eb52a4a84c2385eba095dab967b9704d: Status 404 returned error can't find the container with id 01eea6dc54ac13bbfb07632798289c33eb52a4a84c2385eba095dab967b9704d
Sep 18 18:59:19.206032 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.206006    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"]
Sep 18 18:59:19.208818 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:19.208799    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e369bc_6706_468f_b542_0d7e250c165f.slice/crio-a745d6c3e6f352a41466900275392325a16926c4a0a91a6e4fa5a979e74c679a WatchSource:0}: Error finding container a745d6c3e6f352a41466900275392325a16926c4a0a91a6e4fa5a979e74c679a: Status 404 returned error can't find the container with id a745d6c3e6f352a41466900275392325a16926c4a0a91a6e4fa5a979e74c679a
Sep 18 18:59:19.218644 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.218582    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6" podStartSLOduration=2.802767145 podStartE2EDuration="3.2185635s" totalImagesPullingTime="415.796355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:19.217280833 +0000 UTC m=+883.009957809" watchObservedRunningTime="2026-09-18 18:59:19.2185635 +0000 UTC m=+883.011240430"
Sep 18 18:59:19.303420 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.303391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:19.513993 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:19.513969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:20.186297 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.186262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj" event={"ID":"ae80296c-c554-4668-9e80-2f64e37e81a9","Type":"ContainerStarted","Data":"c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2"}
Sep 18 18:59:20.186736 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.186323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj" event={"ID":"ae80296c-c554-4668-9e80-2f64e37e81a9","Type":"ContainerStarted","Data":"01eea6dc54ac13bbfb07632798289c33eb52a4a84c2385eba095dab967b9704d"}
Sep 18 18:59:20.186736 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.186356    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:20.187919 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.187875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz" event={"ID":"f3e369bc-6706-468f-b542-0d7e250c165f","Type":"ContainerStarted","Data":"2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a"}
Sep 18 18:59:20.188037 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.187938    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:20.188037 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.187955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz" event={"ID":"f3e369bc-6706-468f-b542-0d7e250c165f","Type":"ContainerStarted","Data":"a745d6c3e6f352a41466900275392325a16926c4a0a91a6e4fa5a979e74c679a"}
Sep 18 18:59:20.236190 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.236144    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj" podStartSLOduration=2.236131582 podStartE2EDuration="2.236131582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:20.234470526 +0000 UTC m=+884.027147474" watchObservedRunningTime="2026-09-18 18:59:20.236131582 +0000 UTC m=+884.028808512"
Sep 18 18:59:20.270071 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.270032    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz" podStartSLOduration=2.270017672 podStartE2EDuration="2.270017672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:20.267730254 +0000 UTC m=+884.060407206" watchObservedRunningTime="2026-09-18 18:59:20.270017672 +0000 UTC m=+884.062694660"
Sep 18 18:59:20.868376 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:20.868347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:21.464843 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:21.464815    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:21.816045 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:21.815972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:23.107358 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.107324    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:23.340753 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.340720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:23.518268 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.518236    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"]
Sep 18 18:59:23.522231 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.522211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.525491 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.525465    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3a81e5-dockercfg-t6hgx\""
Sep 18 18:59:23.525579 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.525540    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3a81e5\""
Sep 18 18:59:23.541916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.541875    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"]
Sep 18 18:59:23.618140 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.618112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709e318c-09aa-465d-a108-9cb6833371ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.618234 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.618151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbm6z\" (UniqueName: \"kubernetes.io/projected/709e318c-09aa-465d-a108-9cb6833371ea-kube-api-access-vbm6z\") pod \"payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.719275 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.719247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709e318c-09aa-465d-a108-9cb6833371ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.719359 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.719284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbm6z\" (UniqueName: \"kubernetes.io/projected/709e318c-09aa-465d-a108-9cb6833371ea-kube-api-access-vbm6z\") pod \"payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.719912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.719869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709e318c-09aa-465d-a108-9cb6833371ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.746801 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.746777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbm6z\" (UniqueName: \"kubernetes.io/projected/709e318c-09aa-465d-a108-9cb6833371ea-kube-api-access-vbm6z\") pod \"payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.788324 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.788273    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"]
Sep 18 18:59:23.790915 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.790902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:23.819663 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.819630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5szf5\" (UniqueName: \"kubernetes.io/projected/bfa30389-314c-4369-a0ac-84cbca9a0e5f-kube-api-access-5szf5\") pod \"payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:23.819663 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.819668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa30389-314c-4369-a0ac-84cbca9a0e5f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:23.832549 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.832531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:23.843578 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.843534    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"]
Sep 18 18:59:23.920237 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.920210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5szf5\" (UniqueName: \"kubernetes.io/projected/bfa30389-314c-4369-a0ac-84cbca9a0e5f-kube-api-access-5szf5\") pod \"payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:23.920386 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.920247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa30389-314c-4369-a0ac-84cbca9a0e5f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:23.921058 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.921037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa30389-314c-4369-a0ac-84cbca9a0e5f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:23.937449 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:23.937420    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5szf5\" (UniqueName: \"kubernetes.io/projected/bfa30389-314c-4369-a0ac-84cbca9a0e5f-kube-api-access-5szf5\") pod \"payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:24.100500 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.100437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:24.122504 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.122480    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"]
Sep 18 18:59:24.124298 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:24.124270    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709e318c_09aa_465d_a108_9cb6833371ea.slice/crio-ee49b9ab4de48717d00220c19634a707c57ed844c8accf028b9eb3b8a00ab779 WatchSource:0}: Error finding container ee49b9ab4de48717d00220c19634a707c57ed844c8accf028b9eb3b8a00ab779: Status 404 returned error can't find the container with id ee49b9ab4de48717d00220c19634a707c57ed844c8accf028b9eb3b8a00ab779
Sep 18 18:59:24.215817 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.215782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw" event={"ID":"709e318c-09aa-465d-a108-9cb6833371ea","Type":"ContainerStarted","Data":"675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85"}
Sep 18 18:59:24.215964 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.215833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw" event={"ID":"709e318c-09aa-465d-a108-9cb6833371ea","Type":"ContainerStarted","Data":"ee49b9ab4de48717d00220c19634a707c57ed844c8accf028b9eb3b8a00ab779"}
Sep 18 18:59:24.216028 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.215997    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:24.282638 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.282589    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw" podStartSLOduration=1.282573551 podStartE2EDuration="1.282573551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:24.280972058 +0000 UTC m=+888.073649000" watchObservedRunningTime="2026-09-18 18:59:24.282573551 +0000 UTC m=+888.075250511"
Sep 18 18:59:24.419908 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.419867    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"]
Sep 18 18:59:24.421113 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:24.421089    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa30389_314c_4369_a0ac_84cbca9a0e5f.slice/crio-b7bb983212a7582352f547ab69af4eff0f44600001081304b26a973f19b8fdc9 WatchSource:0}: Error finding container b7bb983212a7582352f547ab69af4eff0f44600001081304b26a973f19b8fdc9: Status 404 returned error can't find the container with id b7bb983212a7582352f547ab69af4eff0f44600001081304b26a973f19b8fdc9
Sep 18 18:59:24.537645 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:24.537618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:25.193146 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:25.193120    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:25.222739 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:25.222700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr" event={"ID":"bfa30389-314c-4369-a0ac-84cbca9a0e5f","Type":"ContainerStarted","Data":"790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0"}
Sep 18 18:59:25.222909 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:25.222759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr" event={"ID":"bfa30389-314c-4369-a0ac-84cbca9a0e5f","Type":"ContainerStarted","Data":"b7bb983212a7582352f547ab69af4eff0f44600001081304b26a973f19b8fdc9"}
Sep 18 18:59:25.265576 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:25.265523    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr" podStartSLOduration=2.265508223 podStartE2EDuration="2.265508223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:25.262066028 +0000 UTC m=+889.054742981" watchObservedRunningTime="2026-09-18 18:59:25.265508223 +0000 UTC m=+889.058185188"
Sep 18 18:59:25.748694 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:25.748664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:26.227419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:26.227392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:26.492710 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:26.492652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:27.710280 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:27.710258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:27.938232 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:27.938207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:28.065511 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:28.065433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:28.235793 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:28.235760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:28.360235 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:28.360165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:29.424413 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:29.424385    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:30.575006 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:30.574973    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:31.195205 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:31.195178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:31.195348 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:31.195270    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:31.933916 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:31.933862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:33.423712 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:33.423682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:34.631369 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:34.631339    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:35.224627 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:35.224598    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 18:59:35.756757 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:35.756727    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:36.200063 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.200030    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 18:59:36.262151 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.262116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:36.543530 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.543447    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"]
Sep 18 18:59:36.550166 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.550144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.553257 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.553234    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-942883\"/\"simulator-w4-942883-kserve-self-signed-certs\""
Sep 18 18:59:36.553363 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.553236    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-942883\"/\"openshift-service-ca.crt\""
Sep 18 18:59:36.553363 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.553235    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-942883\"/\"kube-root-ca.crt\""
Sep 18 18:59:36.553363 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.553328    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-942883\"/\"default-dockercfg-v6rfk\""
Sep 18 18:59:36.573737 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.573705    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"]
Sep 18 18:59:36.612746 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.611039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-dshm\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.612746 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.611093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tmp-dir\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.612746 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.611133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-home\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.612746 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.611174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqr55\" (UniqueName: \"kubernetes.io/projected/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-kube-api-access-qqr55\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.612746 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.611236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tls-certs\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.612746 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.611260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-model-cache\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.711912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.711853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-dshm\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.711912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.711917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tmp-dir\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.712152 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.711952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-home\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.712152 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.711991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqr55\" (UniqueName: \"kubernetes.io/projected/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-kube-api-access-qqr55\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.712152 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.712049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tls-certs\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.712152 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.712071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-model-cache\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.712345 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.712330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-home\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.712437 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.712414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tmp-dir\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.712564 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.712539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-model-cache\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.714453 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.714424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-dshm\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.714629 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.714612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tls-certs\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.740165 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.740135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqr55\" (UniqueName: \"kubernetes.io/projected/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-kube-api-access-qqr55\") pod \"simulator-w4-942883-kserve-777d6c48d7-btv8x\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") " pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:36.808246 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.808176    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:36.859707 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:36.859677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:37.148483 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:37.148455    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"]
Sep 18 18:59:37.150821 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:37.150795    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec3fca7_dafe_4edf_8c2f_6290ba534ab5.slice/crio-9ca24df1835af8a44c296ad081ef00b8f2e8b321ca92bd55622ed7e309c8f097 WatchSource:0}: Error finding container 9ca24df1835af8a44c296ad081ef00b8f2e8b321ca92bd55622ed7e309c8f097: Status 404 returned error can't find the container with id 9ca24df1835af8a44c296ad081ef00b8f2e8b321ca92bd55622ed7e309c8f097
Sep 18 18:59:37.235353 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:37.235322    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 18:59:37.283731 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:37.283696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x" event={"ID":"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5","Type":"ContainerStarted","Data":"eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738"}
Sep 18 18:59:37.283861 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:37.283739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x" event={"ID":"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5","Type":"ContainerStarted","Data":"9ca24df1835af8a44c296ad081ef00b8f2e8b321ca92bd55622ed7e309c8f097"}
Sep 18 18:59:37.342235 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:37.342167    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x" podStartSLOduration=1.342144716 podStartE2EDuration="1.342144716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:37.338627537 +0000 UTC m=+901.131304468" watchObservedRunningTime="2026-09-18 18:59:37.342144716 +0000 UTC m=+901.134821645"
Sep 18 18:59:38.284516 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:38.284488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:38.649968 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:38.649940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:38.840847 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:38.840819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:39.880106 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:39.880076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:40.875165 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:40.875028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:42.457901 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:42.457860    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:43.659108 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:43.659065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:44.746412 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:44.746381    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"]
Sep 18 18:59:44.746789 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:44.746546    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6" podUID="085997cb-cd70-4b5c-91de-598c3590e647" containerName="maas-api" containerID="cri-o://7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545" gracePeriod=30
Sep 18 18:59:44.796950 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:44.796877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:44.863762 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:44.863733    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"]
Sep 18 18:59:44.864003 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:44.863981    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz" podUID="f3e369bc-6706-468f-b542-0d7e250c165f" containerName="payload-processing" containerID="cri-o://2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a" gracePeriod=30
Sep 18 18:59:44.904087 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:44.904057    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"]
Sep 18 18:59:44.904264 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:44.904244    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj" podUID="ae80296c-c554-4668-9e80-2f64e37e81a9" containerName="payload-pre-processing" containerID="cri-o://c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2" gracePeriod=30
Sep 18 18:59:45.024556 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.024532    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:45.083295 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.083272    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-metrics-certs\") pod \"085997cb-cd70-4b5c-91de-598c3590e647\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") "
Sep 18 18:59:45.083458 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.083442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls\") pod \"085997cb-cd70-4b5c-91de-598c3590e647\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") "
Sep 18 18:59:45.083529 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.083474    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/085997cb-cd70-4b5c-91de-598c3590e647-kube-api-access-wkqsb\") pod \"085997cb-cd70-4b5c-91de-598c3590e647\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") "
Sep 18 18:59:45.083584 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.083529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085997cb-cd70-4b5c-91de-598c3590e647-openshift-service-ca\") pod \"085997cb-cd70-4b5c-91de-598c3590e647\" (UID: \"085997cb-cd70-4b5c-91de-598c3590e647\") "
Sep 18 18:59:45.083930 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.083901    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085997cb-cd70-4b5c-91de-598c3590e647-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "085997cb-cd70-4b5c-91de-598c3590e647" (UID: "085997cb-cd70-4b5c-91de-598c3590e647"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:59:45.086318 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.086295    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085997cb-cd70-4b5c-91de-598c3590e647-kube-api-access-wkqsb" (OuterVolumeSpecName: "kube-api-access-wkqsb") pod "085997cb-cd70-4b5c-91de-598c3590e647" (UID: "085997cb-cd70-4b5c-91de-598c3590e647"). InnerVolumeSpecName "kube-api-access-wkqsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:59:45.087912 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.087847    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "085997cb-cd70-4b5c-91de-598c3590e647" (UID: "085997cb-cd70-4b5c-91de-598c3590e647"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:59:45.088006 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.087922    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "085997cb-cd70-4b5c-91de-598c3590e647" (UID: "085997cb-cd70-4b5c-91de-598c3590e647"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 18:59:45.130419 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.130391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:45.156302 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.156279    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:45.163603 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.163583    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:45.185030 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.185006    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-metrics-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.185030 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.185025    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/085997cb-cd70-4b5c-91de-598c3590e647-maas-api-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.185144 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.185033    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqsb\" (UniqueName: \"kubernetes.io/projected/085997cb-cd70-4b5c-91de-598c3590e647-kube-api-access-wkqsb\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.185144 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.185044    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/085997cb-cd70-4b5c-91de-598c3590e647-openshift-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.285591 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.285528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwcn2\" (UniqueName: \"kubernetes.io/projected/f3e369bc-6706-468f-b542-0d7e250c165f-kube-api-access-jwcn2\") pod \"f3e369bc-6706-468f-b542-0d7e250c165f\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") "
Sep 18 18:59:45.285591 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.285578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae80296c-c554-4668-9e80-2f64e37e81a9-plugins-config-volume\") pod \"ae80296c-c554-4668-9e80-2f64e37e81a9\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") "
Sep 18 18:59:45.285755 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.285612    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e369bc-6706-468f-b542-0d7e250c165f-plugins-config-volume\") pod \"f3e369bc-6706-468f-b542-0d7e250c165f\" (UID: \"f3e369bc-6706-468f-b542-0d7e250c165f\") "
Sep 18 18:59:45.285755 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.285641    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8k76\" (UniqueName: \"kubernetes.io/projected/ae80296c-c554-4668-9e80-2f64e37e81a9-kube-api-access-f8k76\") pod \"ae80296c-c554-4668-9e80-2f64e37e81a9\" (UID: \"ae80296c-c554-4668-9e80-2f64e37e81a9\") "
Sep 18 18:59:45.286040 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.286014    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e369bc-6706-468f-b542-0d7e250c165f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3e369bc-6706-468f-b542-0d7e250c165f" (UID: "f3e369bc-6706-468f-b542-0d7e250c165f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:59:45.286153 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.286015    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae80296c-c554-4668-9e80-2f64e37e81a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae80296c-c554-4668-9e80-2f64e37e81a9" (UID: "ae80296c-c554-4668-9e80-2f64e37e81a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 18:59:45.287735 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.287713    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e369bc-6706-468f-b542-0d7e250c165f-kube-api-access-jwcn2" (OuterVolumeSpecName: "kube-api-access-jwcn2") pod "f3e369bc-6706-468f-b542-0d7e250c165f" (UID: "f3e369bc-6706-468f-b542-0d7e250c165f"). InnerVolumeSpecName "kube-api-access-jwcn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:59:45.287803 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.287750    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae80296c-c554-4668-9e80-2f64e37e81a9-kube-api-access-f8k76" (OuterVolumeSpecName: "kube-api-access-f8k76") pod "ae80296c-c554-4668-9e80-2f64e37e81a9" (UID: "ae80296c-c554-4668-9e80-2f64e37e81a9"). InnerVolumeSpecName "kube-api-access-f8k76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 18:59:45.322057 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.322024    2582 generic.go:356] "Generic (PLEG): container finished" podID="085997cb-cd70-4b5c-91de-598c3590e647" containerID="7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545" exitCode=0
Sep 18 18:59:45.322175 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.322066    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"
Sep 18 18:59:45.322175 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.322118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6" event={"ID":"085997cb-cd70-4b5c-91de-598c3590e647","Type":"ContainerDied","Data":"7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545"}
Sep 18 18:59:45.322175 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.322167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6" event={"ID":"085997cb-cd70-4b5c-91de-598c3590e647","Type":"ContainerDied","Data":"25825a9bdddbf2dd5dc180a2c33ab3f67d9af22681ddf62a2104dfafa96627df"}
Sep 18 18:59:45.322331 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.322196    2582 scope.go:117] "RemoveContainer" containerID="7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545"
Sep 18 18:59:45.323564 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.323546    2582 generic.go:356] "Generic (PLEG): container finished" podID="ae80296c-c554-4668-9e80-2f64e37e81a9" containerID="c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2" exitCode=0
Sep 18 18:59:45.323827 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.323572    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"
Sep 18 18:59:45.323827 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.323664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj" event={"ID":"ae80296c-c554-4668-9e80-2f64e37e81a9","Type":"ContainerDied","Data":"c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2"}
Sep 18 18:59:45.323827 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.323702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj" event={"ID":"ae80296c-c554-4668-9e80-2f64e37e81a9","Type":"ContainerDied","Data":"01eea6dc54ac13bbfb07632798289c33eb52a4a84c2385eba095dab967b9704d"}
Sep 18 18:59:45.325244 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.325221    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3e369bc-6706-468f-b542-0d7e250c165f" containerID="2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a" exitCode=0
Sep 18 18:59:45.325385 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.325265    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"
Sep 18 18:59:45.325385 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.325295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz" event={"ID":"f3e369bc-6706-468f-b542-0d7e250c165f","Type":"ContainerDied","Data":"2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a"}
Sep 18 18:59:45.325385 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.325332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz" event={"ID":"f3e369bc-6706-468f-b542-0d7e250c165f","Type":"ContainerDied","Data":"a745d6c3e6f352a41466900275392325a16926c4a0a91a6e4fa5a979e74c679a"}
Sep 18 18:59:45.331933 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.331870    2582 scope.go:117] "RemoveContainer" containerID="7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545"
Sep 18 18:59:45.332230 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:59:45.332213    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545\": container with ID starting with 7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545 not found: ID does not exist" containerID="7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545"
Sep 18 18:59:45.332298 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.332239    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545"} err="failed to get container status \"7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545\": rpc error: code = NotFound desc = could not find container \"7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545\": container with ID starting with 7d10f4a555a8af280e1fc3d895b2e0b0eb0d33829e755798f640e802a52f5545 not found: ID does not exist"
Sep 18 18:59:45.332298 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.332256    2582 scope.go:117] "RemoveContainer" containerID="c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2"
Sep 18 18:59:45.340572 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.340557    2582 scope.go:117] "RemoveContainer" containerID="c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2"
Sep 18 18:59:45.340835 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:59:45.340816    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2\": container with ID starting with c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2 not found: ID does not exist" containerID="c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2"
Sep 18 18:59:45.341002 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.340839    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2"} err="failed to get container status \"c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2\": rpc error: code = NotFound desc = could not find container \"c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2\": container with ID starting with c4d31a8a1dd44949cfdd8f097d33797440e8271028844cfb823175466c3da2b2 not found: ID does not exist"
Sep 18 18:59:45.341002 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.340854    2582 scope.go:117] "RemoveContainer" containerID="2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a"
Sep 18 18:59:45.348345 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.348329    2582 scope.go:117] "RemoveContainer" containerID="2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a"
Sep 18 18:59:45.348594 ip-10-0-132-168 kubenswrapper[2582]: E0918 18:59:45.348577    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a\": container with ID starting with 2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a not found: ID does not exist" containerID="2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a"
Sep 18 18:59:45.348640 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.348601    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a"} err="failed to get container status \"2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a\": rpc error: code = NotFound desc = could not find container \"2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a\": container with ID starting with 2c45dc3a6f6fd8958c30c062fcb40a7e4695aa35ec7ba26b3fddfeb452a7933a not found: ID does not exist"
Sep 18 18:59:45.386525 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.386502    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwcn2\" (UniqueName: \"kubernetes.io/projected/f3e369bc-6706-468f-b542-0d7e250c165f-kube-api-access-jwcn2\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.386619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.386529    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae80296c-c554-4668-9e80-2f64e37e81a9-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.386619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.386544    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3e369bc-6706-468f-b542-0d7e250c165f-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.386619 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.386557    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8k76\" (UniqueName: \"kubernetes.io/projected/ae80296c-c554-4668-9e80-2f64e37e81a9-kube-api-access-f8k76\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 18:59:45.388479 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.388459    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"]
Sep 18 18:59:45.400536 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.400512    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8cbeacf3-75c895895f-nzgb6"]
Sep 18 18:59:45.429643 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.429624    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"]
Sep 18 18:59:45.431413 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.431396    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8cbeacf3-7bbdc97b9d-49slj"]
Sep 18 18:59:45.434984 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.434966    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:45.453595 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.453572    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"]
Sep 18 18:59:45.464745 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.463632    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8cbeacf3-7db456cf65-xlwwz"]
Sep 18 18:59:45.549793 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.549734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:45.654678 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:45.654648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:46.037406 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.037385    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:46.221173 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.221137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:46.335384 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.335325    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085997cb-cd70-4b5c-91de-598c3590e647" path="/var/lib/kubelet/pods/085997cb-cd70-4b5c-91de-598c3590e647/volumes"
Sep 18 18:59:46.335697 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.335684    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae80296c-c554-4668-9e80-2f64e37e81a9" path="/var/lib/kubelet/pods/ae80296c-c554-4668-9e80-2f64e37e81a9/volumes"
Sep 18 18:59:46.336067 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.336036    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3e369bc-6706-468f-b542-0d7e250c165f" path="/var/lib/kubelet/pods/f3e369bc-6706-468f-b542-0d7e250c165f/volumes"
Sep 18 18:59:46.387019 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.386999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:46.516308 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.516286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:46.859979 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.859945    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:46.860135 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.859998    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:46.870104 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.870082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:46.872955 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:46.872938    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:47.238181 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:47.238161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:47.348177 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:47.348159    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 18:59:47.864095 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:47.864066    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:50.024144 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:50.024113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:50.378990 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:50.378916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:50.802720 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:50.802693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:51.882849 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:51.882817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:53.291506 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:53.291479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:54.728037 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:54.728007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:55.998055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:55.998022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:56.170096 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:56.170071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:56.981874 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:56.981846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:57.190086 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.190059    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:57.319104 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319041    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"]
Sep 18 18:59:57.319431 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319416    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae80296c-c554-4668-9e80-2f64e37e81a9" containerName="payload-pre-processing"
Sep 18 18:59:57.319473 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319434    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae80296c-c554-4668-9e80-2f64e37e81a9" containerName="payload-pre-processing"
Sep 18 18:59:57.319473 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319450    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3e369bc-6706-468f-b542-0d7e250c165f" containerName="payload-processing"
Sep 18 18:59:57.319473 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319456    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3e369bc-6706-468f-b542-0d7e250c165f" containerName="payload-processing"
Sep 18 18:59:57.319473 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085997cb-cd70-4b5c-91de-598c3590e647" containerName="maas-api"
Sep 18 18:59:57.319473 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319468    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="085997cb-cd70-4b5c-91de-598c3590e647" containerName="maas-api"
Sep 18 18:59:57.319624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319552    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae80296c-c554-4668-9e80-2f64e37e81a9" containerName="payload-pre-processing"
Sep 18 18:59:57.319624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319560    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3e369bc-6706-468f-b542-0d7e250c165f" containerName="payload-processing"
Sep 18 18:59:57.319624 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.319570    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="085997cb-cd70-4b5c-91de-598c3590e647" containerName="maas-api"
Sep 18 18:59:57.322637 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.322622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.324946 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.324927    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f4465858\""
Sep 18 18:59:57.325061 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.324927    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f4465858-dockercfg-nr6v6\""
Sep 18 18:59:57.347221 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.347200    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"]
Sep 18 18:59:57.379462 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.379433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwpw\" (UniqueName: \"kubernetes.io/projected/c91275e6-d85f-47ca-9cc6-60b76f79ea61-kube-api-access-kbwpw\") pod \"payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") " pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.379582 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.379477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91275e6-d85f-47ca-9cc6-60b76f79ea61-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") " pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.401068 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.401040    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"]
Sep 18 18:59:57.404241 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.404226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.417984 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.417964    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"]
Sep 18 18:59:57.480923 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.480899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwpw\" (UniqueName: \"kubernetes.io/projected/c91275e6-d85f-47ca-9cc6-60b76f79ea61-kube-api-access-kbwpw\") pod \"payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") " pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.481013 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.480927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91275e6-d85f-47ca-9cc6-60b76f79ea61-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") " pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.481191 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.481168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twt76\" (UniqueName: \"kubernetes.io/projected/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-kube-api-access-twt76\") pod \"payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.481309 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.481239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.481704 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.481683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91275e6-d85f-47ca-9cc6-60b76f79ea61-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") " pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.494086 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.494059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwpw\" (UniqueName: \"kubernetes.io/projected/c91275e6-d85f-47ca-9cc6-60b76f79ea61-kube-api-access-kbwpw\") pod \"payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") " pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.581954 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.581877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.582048 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.582005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twt76\" (UniqueName: \"kubernetes.io/projected/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-kube-api-access-twt76\") pod \"payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.582552 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.582532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.594372 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.594352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twt76\" (UniqueName: \"kubernetes.io/projected/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-kube-api-access-twt76\") pod \"payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.631743 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.631725    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:57.713318 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.713284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:57.878337 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.878272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 18:59:57.931405 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:57.931381    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91275e6_d85f_47ca_9cc6_60b76f79ea61.slice/crio-4da97642894041dce6019210d3363068ee757af67261f791b13b7b2bc40d1f58 WatchSource:0}: Error finding container 4da97642894041dce6019210d3363068ee757af67261f791b13b7b2bc40d1f58: Status 404 returned error can't find the container with id 4da97642894041dce6019210d3363068ee757af67261f791b13b7b2bc40d1f58
Sep 18 18:59:57.935934 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:57.935910    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"]
Sep 18 18:59:58.010097 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.010073    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"]
Sep 18 18:59:58.011435 ip-10-0-132-168 kubenswrapper[2582]: W0918 18:59:58.011402    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e63d7d6_98bc_40e4_8c92_93b5e1ce5087.slice/crio-759b1b177ebc6a6cad3294b350f61cac1ab3ded1329317db7d52375fa151a69e WatchSource:0}: Error finding container 759b1b177ebc6a6cad3294b350f61cac1ab3ded1329317db7d52375fa151a69e: Status 404 returned error can't find the container with id 759b1b177ebc6a6cad3294b350f61cac1ab3ded1329317db7d52375fa151a69e
Sep 18 18:59:58.391598 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.391561    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5" event={"ID":"c91275e6-d85f-47ca-9cc6-60b76f79ea61","Type":"ContainerStarted","Data":"a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7"}
Sep 18 18:59:58.392055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.391621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5" event={"ID":"c91275e6-d85f-47ca-9cc6-60b76f79ea61","Type":"ContainerStarted","Data":"4da97642894041dce6019210d3363068ee757af67261f791b13b7b2bc40d1f58"}
Sep 18 18:59:58.392055 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.391658    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 18:59:58.393249 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.393219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt" event={"ID":"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087","Type":"ContainerStarted","Data":"e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b"}
Sep 18 18:59:58.393390 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.393270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt" event={"ID":"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087","Type":"ContainerStarted","Data":"759b1b177ebc6a6cad3294b350f61cac1ab3ded1329317db7d52375fa151a69e"}
Sep 18 18:59:58.393390 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.393381    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 18:59:58.425376 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.425332    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5" podStartSLOduration=1.425318742 podStartE2EDuration="1.425318742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:58.424617442 +0000 UTC m=+922.217294403" watchObservedRunningTime="2026-09-18 18:59:58.425318742 +0000 UTC m=+922.217995672"
Sep 18 18:59:58.450872 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:58.450830    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt" podStartSLOduration=1.450818399 podStartE2EDuration="1.450818399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 18:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 18:59:58.447834545 +0000 UTC m=+922.240511494" watchObservedRunningTime="2026-09-18 18:59:58.450818399 +0000 UTC m=+922.243495329"
Sep 18 18:59:59.201982 ip-10-0-132-168 kubenswrapper[2582]: I0918 18:59:59.201956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:00.265560 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.265531    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"]
Sep 18 19:00:00.269024 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.269005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"
Sep 18 19:00:00.285871 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.285844    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kgn8b\""
Sep 18 19:00:00.288856 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.288835    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"
Sep 18 19:00:00.289966 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.289944    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4"]
Sep 18 19:00:00.293662 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.293647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4"
Sep 18 19:00:00.320567 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.320545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4"
Sep 18 19:00:00.366708 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.366657    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"]
Sep 18 19:00:00.370875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.370587    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"]
Sep 18 19:00:00.375594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.375572    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4"]
Sep 18 19:00:00.375718 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.375678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"
Sep 18 19:00:00.391894 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.391842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"
Sep 18 19:00:00.417520 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.417417    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"]
Sep 18 19:00:00.634102 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:00:00.634071    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f781b3_39f4_4425_89c1_cb50884edc1c.slice/crio-496b4cfce1a87c60337a9867ba0345021165486f6bd75d449a79eb1161849c73 WatchSource:0}: Error finding container 496b4cfce1a87c60337a9867ba0345021165486f6bd75d449a79eb1161849c73: Status 404 returned error can't find the container with id 496b4cfce1a87c60337a9867ba0345021165486f6bd75d449a79eb1161849c73
Sep 18 19:00:00.634514 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.634492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4"]
Sep 18 19:00:00.672545 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.672523    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:00:00.786994 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:00:00.786959    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17850d02_6541_4344_8e94_5dd24a832c03.slice/crio-852c152ba79851e623e4ba2b9ee3ee2e2f765fdd5c2912fb003ee24ba9dd2ae0 WatchSource:0}: Error finding container 852c152ba79851e623e4ba2b9ee3ee2e2f765fdd5c2912fb003ee24ba9dd2ae0: Status 404 returned error can't find the container with id 852c152ba79851e623e4ba2b9ee3ee2e2f765fdd5c2912fb003ee24ba9dd2ae0
Sep 18 19:00:00.799390 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:00:00.799366    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb290be6f_2408_4625_ae33_70277d380e3d.slice/crio-bb4ace22cfbea15fcab0105a4d47a381214bbef22d364389d946cb2bc8d9e52e WatchSource:0}: Error finding container bb4ace22cfbea15fcab0105a4d47a381214bbef22d364389d946cb2bc8d9e52e: Status 404 returned error can't find the container with id bb4ace22cfbea15fcab0105a4d47a381214bbef22d364389d946cb2bc8d9e52e
Sep 18 19:00:00.800661 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.800640    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"]
Sep 18 19:00:00.802728 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:00.802704    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"]
Sep 18 19:00:01.016131 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:01.016107    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:01.408899 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:01.408859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r" event={"ID":"17850d02-6541-4344-8e94-5dd24a832c03","Type":"ContainerStarted","Data":"852c152ba79851e623e4ba2b9ee3ee2e2f765fdd5c2912fb003ee24ba9dd2ae0"}
Sep 18 19:00:01.409995 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:01.409970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm" event={"ID":"b290be6f-2408-4625-ae33-70277d380e3d","Type":"ContainerStarted","Data":"bb4ace22cfbea15fcab0105a4d47a381214bbef22d364389d946cb2bc8d9e52e"}
Sep 18 19:00:01.411028 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:01.411009    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4" event={"ID":"14f781b3-39f4-4425-89c1-cb50884edc1c","Type":"ContainerStarted","Data":"496b4cfce1a87c60337a9867ba0345021165486f6bd75d449a79eb1161849c73"}
Sep 18 19:00:03.424160 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:03.424101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r" event={"ID":"17850d02-6541-4344-8e94-5dd24a832c03","Type":"ContainerStarted","Data":"e3ed5795ba5939b85aa13284027aa6364bcaa5ca3284e6591af74ce0782f8bcb"}
Sep 18 19:00:03.425719 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:03.425695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm" event={"ID":"b290be6f-2408-4625-ae33-70277d380e3d","Type":"ContainerStarted","Data":"d6a5cd110d06f0bb6366956e6acf69cca6feb0e30d3262250709ea1ebc83a521"}
Sep 18 19:00:03.427233 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:03.427199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4" event={"ID":"14f781b3-39f4-4425-89c1-cb50884edc1c","Type":"ContainerStarted","Data":"d2704692de555b9d9129f169b0ceb159822ad73fbb9b4ef9a4f681703b27c585"}
Sep 18 19:00:03.477910 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:03.477853    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r" podStartSLOduration=1.3212091799999999 podStartE2EDuration="3.477839431s" totalImagesPullingTime="2.156630251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:03.475624495 +0000 UTC m=+927.268301443" watchObservedRunningTime="2026-09-18 19:00:03.477839431 +0000 UTC m=+927.270516360"
Sep 18 19:00:03.543931 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:03.543899    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm" podStartSLOduration=1.400312186 podStartE2EDuration="3.543867647s" totalImagesPullingTime="2.143555461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:03.533141912 +0000 UTC m=+927.325818841" watchObservedRunningTime="2026-09-18 19:00:03.543867647 +0000 UTC m=+927.336544576"
Sep 18 19:00:03.584134 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:03.584095    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4" podStartSLOduration=1.279182663 podStartE2EDuration="3.584081865s" totalImagesPullingTime="2.304899202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:03.582724189 +0000 UTC m=+927.375401119" watchObservedRunningTime="2026-09-18 19:00:03.584081865 +0000 UTC m=+927.376758795"
Sep 18 19:00:04.433645 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:04.433610    2582 generic.go:356] "Generic (PLEG): container finished" podID="17850d02-6541-4344-8e94-5dd24a832c03" containerID="e3ed5795ba5939b85aa13284027aa6364bcaa5ca3284e6591af74ce0782f8bcb" exitCode=7
Sep 18 19:00:04.434023 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:04.433693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r" event={"ID":"17850d02-6541-4344-8e94-5dd24a832c03","Type":"ContainerDied","Data":"e3ed5795ba5939b85aa13284027aa6364bcaa5ca3284e6591af74ce0782f8bcb"}
Sep 18 19:00:04.434023 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:04.433811    2582 scope.go:117] "RemoveContainer" containerID="e3ed5795ba5939b85aa13284027aa6364bcaa5ca3284e6591af74ce0782f8bcb"
Sep 18 19:00:04.435348 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:04.435326    2582 generic.go:356] "Generic (PLEG): container finished" podID="b290be6f-2408-4625-ae33-70277d380e3d" containerID="d6a5cd110d06f0bb6366956e6acf69cca6feb0e30d3262250709ea1ebc83a521" exitCode=0
Sep 18 19:00:04.435431 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:04.435404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm" event={"ID":"b290be6f-2408-4625-ae33-70277d380e3d","Type":"ContainerDied","Data":"d6a5cd110d06f0bb6366956e6acf69cca6feb0e30d3262250709ea1ebc83a521"}
Sep 18 19:00:04.436875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:04.436857    2582 generic.go:356] "Generic (PLEG): container finished" podID="14f781b3-39f4-4425-89c1-cb50884edc1c" containerID="d2704692de555b9d9129f169b0ceb159822ad73fbb9b4ef9a4f681703b27c585" exitCode=0
Sep 18 19:00:04.436967 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:04.436907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4" event={"ID":"14f781b3-39f4-4425-89c1-cb50884edc1c","Type":"ContainerDied","Data":"d2704692de555b9d9129f169b0ceb159822ad73fbb9b4ef9a4f681703b27c585"}
Sep 18 19:00:05.444171 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:05.444139    2582 generic.go:356] "Generic (PLEG): container finished" podID="17850d02-6541-4344-8e94-5dd24a832c03" containerID="9efe2c23536bcb837cf30f4ffe1555ce7a2ed756bd1516ab376d353fdd6c5c00" exitCode=0
Sep 18 19:00:05.444553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:05.444221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r" event={"ID":"17850d02-6541-4344-8e94-5dd24a832c03","Type":"ContainerDied","Data":"9efe2c23536bcb837cf30f4ffe1555ce7a2ed756bd1516ab376d353fdd6c5c00"}
Sep 18 19:00:05.444553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:05.444275    2582 scope.go:117] "RemoveContainer" containerID="e3ed5795ba5939b85aa13284027aa6364bcaa5ca3284e6591af74ce0782f8bcb"
Sep 18 19:00:05.598692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:05.598671    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"
Sep 18 19:00:05.600909 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:05.600873    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4"
Sep 18 19:00:06.451492 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:06.451472    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"
Sep 18 19:00:06.451849 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:06.451499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm" event={"ID":"b290be6f-2408-4625-ae33-70277d380e3d","Type":"ContainerDied","Data":"bb4ace22cfbea15fcab0105a4d47a381214bbef22d364389d946cb2bc8d9e52e"}
Sep 18 19:00:06.451849 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:06.451536    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb4ace22cfbea15fcab0105a4d47a381214bbef22d364389d946cb2bc8d9e52e"
Sep 18 19:00:06.452899 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:06.452853    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4"
Sep 18 19:00:06.453003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:06.452847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829300-2kcf4" event={"ID":"14f781b3-39f4-4425-89c1-cb50884edc1c","Type":"ContainerDied","Data":"496b4cfce1a87c60337a9867ba0345021165486f6bd75d449a79eb1161849c73"}
Sep 18 19:00:06.453065 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:06.453006    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496b4cfce1a87c60337a9867ba0345021165486f6bd75d449a79eb1161849c73"
Sep 18 19:00:06.577754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:06.577733    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"
Sep 18 19:00:07.459097 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:07.459073    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"
Sep 18 19:00:07.459452 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:07.459110    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r" event={"ID":"17850d02-6541-4344-8e94-5dd24a832c03","Type":"ContainerDied","Data":"852c152ba79851e623e4ba2b9ee3ee2e2f765fdd5c2912fb003ee24ba9dd2ae0"}
Sep 18 19:00:07.459452 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:07.459146    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="852c152ba79851e623e4ba2b9ee3ee2e2f765fdd5c2912fb003ee24ba9dd2ae0"
Sep 18 19:00:09.400425 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:09.400399    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 19:00:09.400876 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:09.400462    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 19:00:20.614299 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614269    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"]
Sep 18 19:00:20.614634 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614624    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17850d02-6541-4344-8e94-5dd24a832c03" containerName="cleanup"
Sep 18 19:00:20.614683 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614636    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17850d02-6541-4344-8e94-5dd24a832c03" containerName="cleanup"
Sep 18 19:00:20.614683 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614658    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f781b3-39f4-4425-89c1-cb50884edc1c" containerName="cleanup"
Sep 18 19:00:20.614683 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614664    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f781b3-39f4-4425-89c1-cb50884edc1c" containerName="cleanup"
Sep 18 19:00:20.614683 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614673    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b290be6f-2408-4625-ae33-70277d380e3d" containerName="cleanup"
Sep 18 19:00:20.614683 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614679    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b290be6f-2408-4625-ae33-70277d380e3d" containerName="cleanup"
Sep 18 19:00:20.614840 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614734    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b290be6f-2408-4625-ae33-70277d380e3d" containerName="cleanup"
Sep 18 19:00:20.614840 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614742    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="17850d02-6541-4344-8e94-5dd24a832c03" containerName="cleanup"
Sep 18 19:00:20.614840 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614748    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f781b3-39f4-4425-89c1-cb50884edc1c" containerName="cleanup"
Sep 18 19:00:20.614840 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.614754    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="17850d02-6541-4344-8e94-5dd24a832c03" containerName="cleanup"
Sep 18 19:00:20.617727 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.617711    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.621944 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.621913    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a444c\"/\"simulator-w0-1a444c-kserve-self-signed-certs\""
Sep 18 19:00:20.621944 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.621937    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a444c\"/\"openshift-service-ca.crt\""
Sep 18 19:00:20.622122 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.621921    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a444c\"/\"kube-root-ca.crt\""
Sep 18 19:00:20.622122 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.621921    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a444c\"/\"default-dockercfg-c8v6m\""
Sep 18 19:00:20.657022 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.656999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"]
Sep 18 19:00:20.766315 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.766289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-tmp-dir\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.766490 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.766323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e0c692-8dbb-4645-82ca-70ae0429e810-tls-certs\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.766490 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.766349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-home\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.766490 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.766409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-dshm\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.766490 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.766434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-model-cache\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.766643 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.766498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2h9g\" (UniqueName: \"kubernetes.io/projected/60e0c692-8dbb-4645-82ca-70ae0429e810-kube-api-access-d2h9g\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.821426 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.821408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:20.867401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-home\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-dshm\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867557 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-model-cache\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867557 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2h9g\" (UniqueName: \"kubernetes.io/projected/60e0c692-8dbb-4645-82ca-70ae0429e810-kube-api-access-d2h9g\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867557 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-tmp-dir\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867722 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-home\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867783 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e0c692-8dbb-4645-82ca-70ae0429e810-tls-certs\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867840 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-model-cache\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.867918 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.867848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-tmp-dir\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.869736 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.869703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-dshm\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.870041 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.870024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e0c692-8dbb-4645-82ca-70ae0429e810-tls-certs\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.884701 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.884677    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2h9g\" (UniqueName: \"kubernetes.io/projected/60e0c692-8dbb-4645-82ca-70ae0429e810-kube-api-access-d2h9g\") pod \"simulator-w0-1a444c-kserve-57cb6f879b-h27wp\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") " pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:20.926738 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:20.926714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:21.214686 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:21.214661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"]
Sep 18 19:00:21.216529 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:00:21.216502    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e0c692_8dbb_4645_82ca_70ae0429e810.slice/crio-9d2103e7aab7f9a979971b711cf947737565a9d380a0825ee8cd1774dd2bd717 WatchSource:0}: Error finding container 9d2103e7aab7f9a979971b711cf947737565a9d380a0825ee8cd1774dd2bd717: Status 404 returned error can't find the container with id 9d2103e7aab7f9a979971b711cf947737565a9d380a0825ee8cd1774dd2bd717
Sep 18 19:00:21.535985 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:21.535899    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp" event={"ID":"60e0c692-8dbb-4645-82ca-70ae0429e810","Type":"ContainerStarted","Data":"1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020"}
Sep 18 19:00:21.535985 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:21.535945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp" event={"ID":"60e0c692-8dbb-4645-82ca-70ae0429e810","Type":"ContainerStarted","Data":"9d2103e7aab7f9a979971b711cf947737565a9d380a0825ee8cd1774dd2bd717"}
Sep 18 19:00:21.577177 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:21.577129    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp" podStartSLOduration=1.577113051 podStartE2EDuration="1.577113051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:21.573829421 +0000 UTC m=+945.366506352" watchObservedRunningTime="2026-09-18 19:00:21.577113051 +0000 UTC m=+945.369790019"
Sep 18 19:00:24.487910 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:24.487860    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:25.619613 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:25.619584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:26.534673 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:26.534640    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:00:26.763508 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:26.763470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:28.047491 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:28.047461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:29.226958 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:29.226916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:30.692174 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:30.692139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:30.927073 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:30.927034    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:30.927073 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:30.927085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:30.938660 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:30.938638    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:31.509397 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:31.509368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:31.598283 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:31.598260    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:00:32.329177 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:32.329144    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:33.714220 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:33.714193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:34.797938 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:34.797908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:35.908232 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:35.908202    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"]
Sep 18 19:00:35.918754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:35.918727    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f4465858-29829300-xk46r"]
Sep 18 19:00:35.933811 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:35.933784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:35.949326 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:35.949304    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"]
Sep 18 19:00:35.949536 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:35.949516    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5" podUID="c91275e6-d85f-47ca-9cc6-60b76f79ea61" containerName="payload-processing" containerID="cri-o://a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7" gracePeriod=30
Sep 18 19:00:35.974260 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:35.974227    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"]
Sep 18 19:00:35.974441 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:35.974408    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt" podUID="1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" containerName="payload-pre-processing" containerID="cri-o://e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b" gracePeriod=30
Sep 18 19:00:36.099302 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.099279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:36.273379 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.273356    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 19:00:36.285713 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.285684    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbwpw\" (UniqueName: \"kubernetes.io/projected/c91275e6-d85f-47ca-9cc6-60b76f79ea61-kube-api-access-kbwpw\") pod \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") "
Sep 18 19:00:36.285832 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.285733    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91275e6-d85f-47ca-9cc6-60b76f79ea61-plugins-config-volume\") pod \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\" (UID: \"c91275e6-d85f-47ca-9cc6-60b76f79ea61\") "
Sep 18 19:00:36.286081 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.286062    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c91275e6-d85f-47ca-9cc6-60b76f79ea61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c91275e6-d85f-47ca-9cc6-60b76f79ea61" (UID: "c91275e6-d85f-47ca-9cc6-60b76f79ea61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:00:36.287707 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.287688    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91275e6-d85f-47ca-9cc6-60b76f79ea61-kube-api-access-kbwpw" (OuterVolumeSpecName: "kube-api-access-kbwpw") pod "c91275e6-d85f-47ca-9cc6-60b76f79ea61" (UID: "c91275e6-d85f-47ca-9cc6-60b76f79ea61"). InnerVolumeSpecName "kube-api-access-kbwpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:36.291199 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.291154    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 19:00:36.336095 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.336060    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17850d02-6541-4344-8e94-5dd24a832c03" path="/var/lib/kubelet/pods/17850d02-6541-4344-8e94-5dd24a832c03/volumes"
Sep 18 19:00:36.372530 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.372498    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:36.386822 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.386788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twt76\" (UniqueName: \"kubernetes.io/projected/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-kube-api-access-twt76\") pod \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") "
Sep 18 19:00:36.386822 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.386822    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-plugins-config-volume\") pod \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\" (UID: \"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087\") "
Sep 18 19:00:36.387155 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.387123    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbwpw\" (UniqueName: \"kubernetes.io/projected/c91275e6-d85f-47ca-9cc6-60b76f79ea61-kube-api-access-kbwpw\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:00:36.387155 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.387152    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c91275e6-d85f-47ca-9cc6-60b76f79ea61-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:00:36.387313 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.387217    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" (UID: "1e63d7d6-98bc-40e4-8c92-93b5e1ce5087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:00:36.388976 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.388956    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-kube-api-access-twt76" (OuterVolumeSpecName: "kube-api-access-twt76") pod "1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" (UID: "1e63d7d6-98bc-40e4-8c92-93b5e1ce5087"). InnerVolumeSpecName "kube-api-access-twt76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:00:36.488476 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.488412    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twt76\" (UniqueName: \"kubernetes.io/projected/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-kube-api-access-twt76\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:00:36.488476 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.488433    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:00:36.613609 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.613577    2582 generic.go:356] "Generic (PLEG): container finished" podID="1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" containerID="e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b" exitCode=0
Sep 18 19:00:36.613749 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.613650    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"
Sep 18 19:00:36.613749 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.613734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt" event={"ID":"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087","Type":"ContainerDied","Data":"e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b"}
Sep 18 19:00:36.613916 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.613838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt" event={"ID":"1e63d7d6-98bc-40e4-8c92-93b5e1ce5087","Type":"ContainerDied","Data":"759b1b177ebc6a6cad3294b350f61cac1ab3ded1329317db7d52375fa151a69e"}
Sep 18 19:00:36.613916 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.613868    2582 scope.go:117] "RemoveContainer" containerID="e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b"
Sep 18 19:00:36.615566 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.615539    2582 generic.go:356] "Generic (PLEG): container finished" podID="c91275e6-d85f-47ca-9cc6-60b76f79ea61" containerID="a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7" exitCode=0
Sep 18 19:00:36.615672 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.615591    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"
Sep 18 19:00:36.615672 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.615620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5" event={"ID":"c91275e6-d85f-47ca-9cc6-60b76f79ea61","Type":"ContainerDied","Data":"a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7"}
Sep 18 19:00:36.615672 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.615670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5" event={"ID":"c91275e6-d85f-47ca-9cc6-60b76f79ea61","Type":"ContainerDied","Data":"4da97642894041dce6019210d3363068ee757af67261f791b13b7b2bc40d1f58"}
Sep 18 19:00:36.624680 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.624663    2582 scope.go:117] "RemoveContainer" containerID="e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b"
Sep 18 19:00:36.624990 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:00:36.624969    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b\": container with ID starting with e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b not found: ID does not exist" containerID="e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b"
Sep 18 19:00:36.625064 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.625000    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b"} err="failed to get container status \"e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b\": rpc error: code = NotFound desc = could not find container \"e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b\": container with ID starting with e668577dab5881e397577463551481b27791f96681fc919e1de68f92b73ced2b not found: ID does not exist"
Sep 18 19:00:36.625064 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.625023    2582 scope.go:117] "RemoveContainer" containerID="a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7"
Sep 18 19:00:36.631237 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.631214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:36.633336 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.633319    2582 scope.go:117] "RemoveContainer" containerID="a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7"
Sep 18 19:00:36.633594 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:00:36.633576    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7\": container with ID starting with a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7 not found: ID does not exist" containerID="a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7"
Sep 18 19:00:36.633653 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.633603    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7"} err="failed to get container status \"a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7\": rpc error: code = NotFound desc = could not find container \"a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7\": container with ID starting with a24fb2976c8c29a838483792d275b7cc4a52043dc0f3d75215ac9bfc892f71a7 not found: ID does not exist"
Sep 18 19:00:36.680229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.680205    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"]
Sep 18 19:00:36.690711 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.690688    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f4465858-5d7655b65d-8sfj5"]
Sep 18 19:00:36.725642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.725614    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"]
Sep 18 19:00:36.727311 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.727290    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f4465858-668c767b7d-tpnjt"]
Sep 18 19:00:36.867080 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:36.867023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:37.070146 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:37.070076    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:37.170197 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:37.170156    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:37.681181 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:37.681130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:37.977227 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:37.977134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:38.335426 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:38.335354    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" path="/var/lib/kubelet/pods/1e63d7d6-98bc-40e4-8c92-93b5e1ce5087/volumes"
Sep 18 19:00:38.335747 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:38.335673    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c91275e6-d85f-47ca-9cc6-60b76f79ea61" path="/var/lib/kubelet/pods/c91275e6-d85f-47ca-9cc6-60b76f79ea61/volumes"
Sep 18 19:00:38.672213 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:38.672184    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:40.080421 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:40.080397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:40.750994 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:40.750967    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:40.976243 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:40.976221    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:41.533007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:41.532986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:43.084925 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:43.084869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:44.583822 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:44.583787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:45.474566 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:45.474512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:45.577823 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:45.577784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:45.679190 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:45.679161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:46.247868 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.247830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:46.407643 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.407619    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:46.506634 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.506563    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"]
Sep 18 19:00:46.506983 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.506967    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c91275e6-d85f-47ca-9cc6-60b76f79ea61" containerName="payload-processing"
Sep 18 19:00:46.507048 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.506986    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c91275e6-d85f-47ca-9cc6-60b76f79ea61" containerName="payload-processing"
Sep 18 19:00:46.507048 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.506995    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" containerName="payload-pre-processing"
Sep 18 19:00:46.507048 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.507000    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" containerName="payload-pre-processing"
Sep 18 19:00:46.507048 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.507015    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17850d02-6541-4344-8e94-5dd24a832c03" containerName="cleanup"
Sep 18 19:00:46.507048 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.507020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17850d02-6541-4344-8e94-5dd24a832c03" containerName="cleanup"
Sep 18 19:00:46.507198 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.507082    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e63d7d6-98bc-40e4-8c92-93b5e1ce5087" containerName="payload-pre-processing"
Sep 18 19:00:46.507198 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.507093    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c91275e6-d85f-47ca-9cc6-60b76f79ea61" containerName="payload-processing"
Sep 18 19:00:46.510164 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.510149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.522870 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.522846    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-980e4a32-dockercfg-w6tv2\""
Sep 18 19:00:46.524037 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.524022    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-980e4a32\""
Sep 18 19:00:46.553590 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.553568    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"]
Sep 18 19:00:46.566507 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.566487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmtv\" (UniqueName: \"kubernetes.io/projected/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-kube-api-access-7mmtv\") pod \"payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.566590 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.566526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.575012 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.574988    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"]
Sep 18 19:00:46.577283 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.577269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:46.600127 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.600103    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"]
Sep 18 19:00:46.667309 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.667272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmtv\" (UniqueName: \"kubernetes.io/projected/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-kube-api-access-7mmtv\") pod \"payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.667309 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.667311    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-plugins-config-volume\") pod \"payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") " pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:46.667509 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.667344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.667509 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.667364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhqb\" (UniqueName: \"kubernetes.io/projected/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-kube-api-access-wxhqb\") pod \"payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") " pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:46.667981 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.667963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.677839 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.677817    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmtv\" (UniqueName: \"kubernetes.io/projected/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-kube-api-access-7mmtv\") pod \"payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.768728 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.768671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-plugins-config-volume\") pod \"payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") " pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:46.768728 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.768723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhqb\" (UniqueName: \"kubernetes.io/projected/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-kube-api-access-wxhqb\") pod \"payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") " pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:46.769275 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.769257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-plugins-config-volume\") pod \"payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") " pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:46.785397 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.785374    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhqb\" (UniqueName: \"kubernetes.io/projected/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-kube-api-access-wxhqb\") pod \"payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") " pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:46.819376 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.819354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:46.886595 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:46.886572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:47.117250 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.117221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"]
Sep 18 19:00:47.117988 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:00:47.117962    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3b8b2b_fa0d_4b50_9bf4_f09d738b2146.slice/crio-b4d18cd5533e43db5e5c77522f2a450798040a11f974eba098421dc2aee85473 WatchSource:0}: Error finding container b4d18cd5533e43db5e5c77522f2a450798040a11f974eba098421dc2aee85473: Status 404 returned error can't find the container with id b4d18cd5533e43db5e5c77522f2a450798040a11f974eba098421dc2aee85473
Sep 18 19:00:47.184437 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.184405    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"]
Sep 18 19:00:47.185424 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:00:47.185398    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc451b7b2_c4e1_46e7_bd6b_ac9091c068bc.slice/crio-99ac9d82278a91f6f27787571996b9e9882ed7d55645cfd374c37f9c12e2b4e0 WatchSource:0}: Error finding container 99ac9d82278a91f6f27787571996b9e9882ed7d55645cfd374c37f9c12e2b4e0: Status 404 returned error can't find the container with id 99ac9d82278a91f6f27787571996b9e9882ed7d55645cfd374c37f9c12e2b4e0
Sep 18 19:00:47.239855 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.239833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:47.674143 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.674108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx" event={"ID":"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc","Type":"ContainerStarted","Data":"19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8"}
Sep 18 19:00:47.674321 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.674262    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:47.674321 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.674303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx" event={"ID":"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc","Type":"ContainerStarted","Data":"99ac9d82278a91f6f27787571996b9e9882ed7d55645cfd374c37f9c12e2b4e0"}
Sep 18 19:00:47.675778 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.675752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5" event={"ID":"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146","Type":"ContainerStarted","Data":"3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703"}
Sep 18 19:00:47.675919 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.675798    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5" event={"ID":"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146","Type":"ContainerStarted","Data":"b4d18cd5533e43db5e5c77522f2a450798040a11f974eba098421dc2aee85473"}
Sep 18 19:00:47.675991 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.675912    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:00:47.704271 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.704234    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx" podStartSLOduration=1.704222433 podStartE2EDuration="1.704222433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:47.704180939 +0000 UTC m=+971.496857868" watchObservedRunningTime="2026-09-18 19:00:47.704222433 +0000 UTC m=+971.496899406"
Sep 18 19:00:47.757798 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:47.757755    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5" podStartSLOduration=1.757739757 podStartE2EDuration="1.757739757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:00:47.753139989 +0000 UTC m=+971.545816936" watchObservedRunningTime="2026-09-18 19:00:47.757739757 +0000 UTC m=+971.550416693"
Sep 18 19:00:48.598402 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:48.598374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:49.762801 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:49.762765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:50.656935 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:50.656911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:00:51.559168 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:51.559132    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:00:58.683438 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:58.683409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:00:58.683946 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:00:58.683479    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:01:04.574250 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:04.574218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:06.111215 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:06.111180    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:07.960642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:07.960612    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:09.155231 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:09.155203    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:10.582341 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.582310    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:10.598776 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.598750    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"]
Sep 18 19:01:10.601523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.601508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.609020 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.609002    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-15c3ed\"/\"openshift-service-ca.crt\""
Sep 18 19:01:10.609374 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.609361    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-15c3ed\"/\"kube-root-ca.crt\""
Sep 18 19:01:10.613348 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.613331    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-15c3ed\"/\"default-dockercfg-6cmfk\""
Sep 18 19:01:10.619173 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.619158    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-15c3ed\"/\"test-model-15c3ed-kserve-self-signed-certs\""
Sep 18 19:01:10.645378 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.645351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227e5808-19aa-4e62-a215-95ab04c056c1-tls-certs\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.645476 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.645395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-dshm\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.645476 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.645423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-home\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.645476 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.645437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-model-cache\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.645580 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.645500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-tmp-dir\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.645580 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.645517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sb95\" (UniqueName: \"kubernetes.io/projected/227e5808-19aa-4e62-a215-95ab04c056c1-kube-api-access-4sb95\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.654218 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.654190    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"]
Sep 18 19:01:10.746066 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227e5808-19aa-4e62-a215-95ab04c056c1-tls-certs\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746157 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-dshm\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746157 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-home\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746270 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-model-cache\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746325 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-tmp-dir\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746379 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sb95\" (UniqueName: \"kubernetes.io/projected/227e5808-19aa-4e62-a215-95ab04c056c1-kube-api-access-4sb95\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746379 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-home\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746533 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-model-cache\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.746651 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.746635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-tmp-dir\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.748391 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.748372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-dshm\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.748653 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.748638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227e5808-19aa-4e62-a215-95ab04c056c1-tls-certs\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.765160 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.765138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sb95\" (UniqueName: \"kubernetes.io/projected/227e5808-19aa-4e62-a215-95ab04c056c1-kube-api-access-4sb95\") pod \"test-model-15c3ed-kserve-7cdd457899-x4b9c\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") " pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:10.852200 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.852142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:10.910751 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:10.910731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:11.233132 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:11.233103    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"]
Sep 18 19:01:11.234223 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:01:11.234185    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod227e5808_19aa_4e62_a215_95ab04c056c1.slice/crio-025a25182d3e8c3603fb639227bbc31bf49cef537382aad084f8e9ae58e95f6b WatchSource:0}: Error finding container 025a25182d3e8c3603fb639227bbc31bf49cef537382aad084f8e9ae58e95f6b: Status 404 returned error can't find the container with id 025a25182d3e8c3603fb639227bbc31bf49cef537382aad084f8e9ae58e95f6b
Sep 18 19:01:11.797207 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:11.797176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c" event={"ID":"227e5808-19aa-4e62-a215-95ab04c056c1","Type":"ContainerStarted","Data":"847d6b70a9028238df5cc7ffe3e4e9a136ebfa438580e61f04256cc82a874a0e"}
Sep 18 19:01:11.797561 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:11.797225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c" event={"ID":"227e5808-19aa-4e62-a215-95ab04c056c1","Type":"ContainerStarted","Data":"025a25182d3e8c3603fb639227bbc31bf49cef537382aad084f8e9ae58e95f6b"}
Sep 18 19:01:11.833814 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:11.833769    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c" podStartSLOduration=1.833756595 podStartE2EDuration="1.833756595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:01:11.832208988 +0000 UTC m=+995.624885929" watchObservedRunningTime="2026-09-18 19:01:11.833756595 +0000 UTC m=+995.626433524"
Sep 18 19:01:12.075684 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:12.075626    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:13.124953 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:13.124921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:14.159397 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:14.159364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:14.993025 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:14.992990    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:15.816506 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:15.816476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:16.631776 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:16.631750    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:17.239539 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:17.239496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:01:17.695458 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:17.695422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:19.479223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:19.479188    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:19.582825 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:19.582800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:20.375970 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:20.375937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:20.674107 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:20.674072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:20.911836 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:20.911801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:20.912007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:20.911848    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:20.923761 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:20.923735    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:20.940440 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:20.940387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:21.208304 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:21.208247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:21.759033 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:21.759010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:21.859634 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:21.859612    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:01:23.278045 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:23.278018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:24.815502 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:24.815468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:26.083101 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:26.083070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:26.425561 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:26.425537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:27.316244 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:27.316213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:28.594531 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:28.594500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:29.425967 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.425936    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"]
Sep 18 19:01:29.426342 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.426122    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx" podUID="c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" containerName="payload-processing" containerID="cri-o://19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8" gracePeriod=30
Sep 18 19:01:29.456508 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.456479    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"]
Sep 18 19:01:29.456702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.456679    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5" podUID="6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" containerName="payload-pre-processing" containerID="cri-o://3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703" gracePeriod=30
Sep 18 19:01:29.722443 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.722423    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:01:29.725766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.725748    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:01:29.891047 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.891021    2582 generic.go:356] "Generic (PLEG): container finished" podID="c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" containerID="19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8" exitCode=0
Sep 18 19:01:29.891181 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.891057    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"
Sep 18 19:01:29.891181 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.891109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx" event={"ID":"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc","Type":"ContainerDied","Data":"19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8"}
Sep 18 19:01:29.891181 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.891158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx" event={"ID":"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc","Type":"ContainerDied","Data":"99ac9d82278a91f6f27787571996b9e9882ed7d55645cfd374c37f9c12e2b4e0"}
Sep 18 19:01:29.891310 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.891187    2582 scope.go:117] "RemoveContainer" containerID="19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8"
Sep 18 19:01:29.892438 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.892418    2582 generic.go:356] "Generic (PLEG): container finished" podID="6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" containerID="3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703" exitCode=0
Sep 18 19:01:29.892554 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.892451    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"
Sep 18 19:01:29.892554 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.892473    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5" event={"ID":"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146","Type":"ContainerDied","Data":"3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703"}
Sep 18 19:01:29.892554 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.892503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5" event={"ID":"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146","Type":"ContainerDied","Data":"b4d18cd5533e43db5e5c77522f2a450798040a11f974eba098421dc2aee85473"}
Sep 18 19:01:29.899978 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.899961    2582 scope.go:117] "RemoveContainer" containerID="19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8"
Sep 18 19:01:29.900245 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:01:29.900228    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8\": container with ID starting with 19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8 not found: ID does not exist" containerID="19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8"
Sep 18 19:01:29.900296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.900257    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8"} err="failed to get container status \"19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8\": rpc error: code = NotFound desc = could not find container \"19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8\": container with ID starting with 19e3c65c319d8d333b0252dd7c9109e801d23221245bf87092ea1c376be04ee8 not found: ID does not exist"
Sep 18 19:01:29.900296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.900275    2582 scope.go:117] "RemoveContainer" containerID="3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703"
Sep 18 19:01:29.905339 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.905320    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-plugins-config-volume\") pod \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") "
Sep 18 19:01:29.905428 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.905361    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxhqb\" (UniqueName: \"kubernetes.io/projected/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-kube-api-access-wxhqb\") pod \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\" (UID: \"c451b7b2-c4e1-46e7-bd6b-ac9091c068bc\") "
Sep 18 19:01:29.905476 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.905428    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-plugins-config-volume\") pod \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") "
Sep 18 19:01:29.905509 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.905476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mmtv\" (UniqueName: \"kubernetes.io/projected/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-kube-api-access-7mmtv\") pod \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\" (UID: \"6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146\") "
Sep 18 19:01:29.905764 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.905738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" (UID: "c451b7b2-c4e1-46e7-bd6b-ac9091c068bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:01:29.905859 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.905831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" (UID: "6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:01:29.907525 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.907495    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-kube-api-access-7mmtv" (OuterVolumeSpecName: "kube-api-access-7mmtv") pod "6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" (UID: "6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146"). InnerVolumeSpecName "kube-api-access-7mmtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:01:29.907525 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.907507    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-kube-api-access-wxhqb" (OuterVolumeSpecName: "kube-api-access-wxhqb") pod "c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" (UID: "c451b7b2-c4e1-46e7-bd6b-ac9091c068bc"). InnerVolumeSpecName "kube-api-access-wxhqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:01:29.910512 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.910493    2582 scope.go:117] "RemoveContainer" containerID="3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703"
Sep 18 19:01:29.910933 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:01:29.910914    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703\": container with ID starting with 3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703 not found: ID does not exist" containerID="3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703"
Sep 18 19:01:29.911006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:29.910939    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703"} err="failed to get container status \"3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703\": rpc error: code = NotFound desc = could not find container \"3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703\": container with ID starting with 3e9bfcd9b9842a804246eb008a5b06830e43a67f903786a6bbf4f1be65241703 not found: ID does not exist"
Sep 18 19:01:30.007274 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.007214    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:01:30.007274 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.007256    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmtv\" (UniqueName: \"kubernetes.io/projected/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146-kube-api-access-7mmtv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:01:30.007274 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.007271    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:01:30.007461 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.007285    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxhqb\" (UniqueName: \"kubernetes.io/projected/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc-kube-api-access-wxhqb\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:01:30.220712 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.220687    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"]
Sep 18 19:01:30.232645 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.232623    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-980e4a32-64d68f88d7-qn8rx"]
Sep 18 19:01:30.240367 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.240344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:30.283232 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.283175    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"]
Sep 18 19:01:30.297559 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.297540    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-980e4a32-b5b757c66-f9wm5"]
Sep 18 19:01:30.335151 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.335131    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" path="/var/lib/kubelet/pods/6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146/volumes"
Sep 18 19:01:30.335454 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:30.335442    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" path="/var/lib/kubelet/pods/c451b7b2-c4e1-46e7-bd6b-ac9091c068bc/volumes"
Sep 18 19:01:31.410073 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:31.410041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:31.673875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:31.673807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:31.997059 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:31.996999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:32.529986 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:32.529956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:33.098193 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:33.098164    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:34.885724 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:34.885693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:36.000922 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:36.000873    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:36.241445 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:36.241414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:37.516805 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:37.516771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:39.888207 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:39.888176    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:40.065269 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:40.065240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:01:42.062985 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:42.062949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:43.225184 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.225154    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:43.589489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.589407    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"]
Sep 18 19:01:43.589861 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.589841    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" containerName="payload-pre-processing"
Sep 18 19:01:43.589989 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.589865    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" containerName="payload-pre-processing"
Sep 18 19:01:43.589989 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.589919    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" containerName="payload-processing"
Sep 18 19:01:43.589989 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.589929    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" containerName="payload-processing"
Sep 18 19:01:43.590143 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.590020    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c451b7b2-c4e1-46e7-bd6b-ac9091c068bc" containerName="payload-processing"
Sep 18 19:01:43.590143 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.590035    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3b8b2b-fa0d-4b50-9bf4-f09d738b2146" containerName="payload-pre-processing"
Sep 18 19:01:43.593447 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.593427    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:43.597917 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:01:43.597866    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="cdb4885c-aad8-4020-ad97-b07b6ae09b62" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:43.598007 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:01:43.597918    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-derive-3341b20d\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3341b20d\"" type="*v1.ConfigMap"
Sep 18 19:01:43.613670 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.613648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume\") pod \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") " pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:43.613772 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.613719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8j69\" (UniqueName: \"kubernetes.io/projected/cdb4885c-aad8-4020-ad97-b07b6ae09b62-kube-api-access-q8j69\") pod \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") " pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:43.658294 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.658273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"]
Sep 18 19:01:43.664972 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.664947    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:43.714673 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.714649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume\") pod \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") " pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:43.714755 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.714702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8j69\" (UniqueName: \"kubernetes.io/projected/cdb4885c-aad8-4020-ad97-b07b6ae09b62-kube-api-access-q8j69\") pod \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") " pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:43.758733 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:43.758712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8j69\" (UniqueName: \"kubernetes.io/projected/cdb4885c-aad8-4020-ad97-b07b6ae09b62-kube-api-access-q8j69\") pod \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") " pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:44.455388 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:44.455363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:44.715841 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:01:44.715750    2582 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-derive-3341b20d: failed to sync configmap cache: timed out waiting for the condition
Sep 18 19:01:44.715841 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:01:44.715804    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume podName:cdb4885c-aad8-4020-ad97-b07b6ae09b62 nodeName:}" failed. No retries permitted until 2026-09-18 19:01:45.215791391 +0000 UTC m=+1029.008468299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume") pod "payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs" (UID: "cdb4885c-aad8-4020-ad97-b07b6ae09b62") : failed to sync configmap cache: timed out waiting for the condition
Sep 18 19:01:44.746108 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:44.746083    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3341b20d\""
Sep 18 19:01:45.224805 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:45.224784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume\") pod \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") " pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:45.225369 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:45.225352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume\") pod \"payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") " pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:45.403052 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:45.403022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:45.710369 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:45.710344    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"]
Sep 18 19:01:45.710696 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:01:45.710553    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb4885c_aad8_4020_ad97_b07b6ae09b62.slice/crio-2dbbab5ee3a022f3d8f01013d0d2c3b60abfa2f98081b6949c369997953c3bfa WatchSource:0}: Error finding container 2dbbab5ee3a022f3d8f01013d0d2c3b60abfa2f98081b6949c369997953c3bfa: Status 404 returned error can't find the container with id 2dbbab5ee3a022f3d8f01013d0d2c3b60abfa2f98081b6949c369997953c3bfa
Sep 18 19:01:45.973306 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:45.973226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs" event={"ID":"cdb4885c-aad8-4020-ad97-b07b6ae09b62","Type":"ContainerStarted","Data":"131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1"}
Sep 18 19:01:45.973306 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:45.973280    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs" event={"ID":"cdb4885c-aad8-4020-ad97-b07b6ae09b62","Type":"ContainerStarted","Data":"2dbbab5ee3a022f3d8f01013d0d2c3b60abfa2f98081b6949c369997953c3bfa"}
Sep 18 19:01:45.973463 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:45.973312    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:46.022781 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:46.022755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:47.774300 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:47.774234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:49.865906 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:49.865805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:52.669722 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:52.669663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:53.559076 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:53.559044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:54.437813 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:54.437763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:55.447613 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:55.447587    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:56.656022 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:56.655984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:56.980094 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:56.980034    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:01:56.996791 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:56.996747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs" podStartSLOduration=13.996734258 podStartE2EDuration="13.996734258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:01:46.035984967 +0000 UTC m=+1029.828661915" watchObservedRunningTime="2026-09-18 19:01:56.996734258 +0000 UTC m=+1040.789411188"
Sep 18 19:01:57.426905 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:57.426836    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:58.185979 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:58.185948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:01:59.056429 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:01:59.056404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:00.055452 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:00.055424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:00.220715 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:00.220686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:00.951535 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:00.951508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:02.400209 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:02.400176    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:02:03.869026 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:03.868999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:04.204535 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:04.204506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:04.650305 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:04.650279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:05.047085 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:05.047023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:05.284535 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:05.284505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:05.565967 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:05.565936    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:06.223060 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.223029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:06.260401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.260372    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"]
Sep 18 19:02:06.260668 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.260644    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs" podUID="cdb4885c-aad8-4020-ad97-b07b6ae09b62" containerName="payload-processing" containerID="cri-o://131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1" gracePeriod=30
Sep 18 19:02:06.342778 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.342754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:06.484250 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.484181    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:06.510396 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.510373    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:02:06.589627 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.589597    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8j69\" (UniqueName: \"kubernetes.io/projected/cdb4885c-aad8-4020-ad97-b07b6ae09b62-kube-api-access-q8j69\") pod \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") "
Sep 18 19:02:06.589750 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.589733    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume\") pod \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\" (UID: \"cdb4885c-aad8-4020-ad97-b07b6ae09b62\") "
Sep 18 19:02:06.590087 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.590066    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdb4885c-aad8-4020-ad97-b07b6ae09b62" (UID: "cdb4885c-aad8-4020-ad97-b07b6ae09b62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:02:06.591816 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.591799    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb4885c-aad8-4020-ad97-b07b6ae09b62-kube-api-access-q8j69" (OuterVolumeSpecName: "kube-api-access-q8j69") pod "cdb4885c-aad8-4020-ad97-b07b6ae09b62" (UID: "cdb4885c-aad8-4020-ad97-b07b6ae09b62"). InnerVolumeSpecName "kube-api-access-q8j69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:02:06.646122 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.646099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:06.690974 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.690950    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4885c-aad8-4020-ad97-b07b6ae09b62-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:02:06.690974 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.690970    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8j69\" (UniqueName: \"kubernetes.io/projected/cdb4885c-aad8-4020-ad97-b07b6ae09b62-kube-api-access-q8j69\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:02:06.871634 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:06.871578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:07.087669 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.087635    2582 generic.go:356] "Generic (PLEG): container finished" podID="cdb4885c-aad8-4020-ad97-b07b6ae09b62" containerID="131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1" exitCode=0
Sep 18 19:02:07.087796 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.087683    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"
Sep 18 19:02:07.087796 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.087722    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs" event={"ID":"cdb4885c-aad8-4020-ad97-b07b6ae09b62","Type":"ContainerDied","Data":"131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1"}
Sep 18 19:02:07.087796 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.087775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs" event={"ID":"cdb4885c-aad8-4020-ad97-b07b6ae09b62","Type":"ContainerDied","Data":"2dbbab5ee3a022f3d8f01013d0d2c3b60abfa2f98081b6949c369997953c3bfa"}
Sep 18 19:02:07.087944 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.087798    2582 scope.go:117] "RemoveContainer" containerID="131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1"
Sep 18 19:02:07.096857 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.096836    2582 scope.go:117] "RemoveContainer" containerID="131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1"
Sep 18 19:02:07.097133 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:02:07.097115    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1\": container with ID starting with 131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1 not found: ID does not exist" containerID="131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1"
Sep 18 19:02:07.097182 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.097141    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1"} err="failed to get container status \"131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1\": rpc error: code = NotFound desc = could not find container \"131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1\": container with ID starting with 131f57e788ce481be2871e4e82d934628dfcc6ff82d2cfeecdac51fa549626b1 not found: ID does not exist"
Sep 18 19:02:07.105472 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.105453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:07.135352 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.135306    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"]
Sep 18 19:02:07.141163 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.141144    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3341b20d-7c988b4657-vxkgs"]
Sep 18 19:02:07.568874 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.568849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:07.916996 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:07.916971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:08.084107 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:08.084088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:08.334740 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:08.334686    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb4885c-aad8-4020-ad97-b07b6ae09b62" path="/var/lib/kubelet/pods/cdb4885c-aad8-4020-ad97-b07b6ae09b62/volumes"
Sep 18 19:02:09.171490 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:09.171468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:09.500525 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:09.500464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:10.376766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:10.376737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:11.319458 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:11.319426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:12.171164 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:12.171135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:13.582792 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:13.582755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:13.703500 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:13.703463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:13.927401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:13.927376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:14.029121 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:14.029094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:14.425086 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:14.425058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:14.529205 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:14.529177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:16.898470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:16.898438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:17.149623 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:17.149552    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:17.262924 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:17.262904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:17.713399 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:17.713377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:23.790044 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:23.789966    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:02:25.334947 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:25.334914    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:34.291056 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.291022    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"]
Sep 18 19:02:34.291553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.291406    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb4885c-aad8-4020-ad97-b07b6ae09b62" containerName="payload-processing"
Sep 18 19:02:34.291553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.291417    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb4885c-aad8-4020-ad97-b07b6ae09b62" containerName="payload-processing"
Sep 18 19:02:34.291553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.291478    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb4885c-aad8-4020-ad97-b07b6ae09b62" containerName="payload-processing"
Sep 18 19:02:34.293838 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.293816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.296317 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.296288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a444c\"/\"premium-w0-1a444c-kserve-self-signed-certs\""
Sep 18 19:02:34.304908 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.304870    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"]
Sep 18 19:02:34.387194 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.387152    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:34.399355 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.399322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-home\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.399496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.399364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhtmv\" (UniqueName: \"kubernetes.io/projected/adcde6df-f011-4e08-95a3-864c9cd8f0a3-kube-api-access-lhtmv\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.399496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.399390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-dshm\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.399496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.399413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-model-cache\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.399496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.399442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tls-certs\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.399694 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.399500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tmp-dir\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501063 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-home\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501257 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501099    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtmv\" (UniqueName: \"kubernetes.io/projected/adcde6df-f011-4e08-95a3-864c9cd8f0a3-kube-api-access-lhtmv\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501257 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-dshm\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501257 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-model-cache\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501257 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tls-certs\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501468 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tmp-dir\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501468 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-home\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501582 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-model-cache\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.501669 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.501647    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tmp-dir\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.503595 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.503572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-dshm\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.503814 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.503795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tls-certs\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.512393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.512362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtmv\" (UniqueName: \"kubernetes.io/projected/adcde6df-f011-4e08-95a3-864c9cd8f0a3-kube-api-access-lhtmv\") pod \"premium-w0-1a444c-kserve-7448b6ddd5-vlhf8\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") " pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.605364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.605278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:34.892545 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:34.892520    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"]
Sep 18 19:02:34.894699 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:02:34.894655    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadcde6df_f011_4e08_95a3_864c9cd8f0a3.slice/crio-7a279663fc4669cac796d517f829bef140db9ce5d820dd4fd55f6dc76a414d34 WatchSource:0}: Error finding container 7a279663fc4669cac796d517f829bef140db9ce5d820dd4fd55f6dc76a414d34: Status 404 returned error can't find the container with id 7a279663fc4669cac796d517f829bef140db9ce5d820dd4fd55f6dc76a414d34
Sep 18 19:02:35.236515 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:35.236415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8" event={"ID":"adcde6df-f011-4e08-95a3-864c9cd8f0a3","Type":"ContainerStarted","Data":"a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a"}
Sep 18 19:02:35.236515 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:35.236471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8" event={"ID":"adcde6df-f011-4e08-95a3-864c9cd8f0a3","Type":"ContainerStarted","Data":"7a279663fc4669cac796d517f829bef140db9ce5d820dd4fd55f6dc76a414d34"}
Sep 18 19:02:35.281013 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:35.280961    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8" podStartSLOduration=1.280944634 podStartE2EDuration="1.280944634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:02:35.28041646 +0000 UTC m=+1079.073093410" watchObservedRunningTime="2026-09-18 19:02:35.280944634 +0000 UTC m=+1079.073621564"
Sep 18 19:02:37.311210 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:37.311176    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:38.057863 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:38.057827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:38.583525 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:38.583493    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:39.521802 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:39.521777    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:40.560577 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:40.560542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:41.339534 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:41.339503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:42.141154 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:42.141126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:43.785328 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:43.785299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:44.605392 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:44.605365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:44.605537 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:44.605408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:44.617266 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:44.617241    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:45.100581 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:45.100536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:02:45.296497 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:45.296467    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:02:50.039377 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:50.039345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:50.212344 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:50.212315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:50.331026 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:50.330961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:02:50.979458 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:02:50.979431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:06.212577 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:06.212543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:10.692631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:10.692596    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:03:12.264560 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:12.264527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:12.378699 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:12.378672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:12.478395 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:12.478370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:22.996582 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:22.996551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:23.857957 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:23.857928    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:29.959852 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:29.959820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:30.070229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:30.070205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:30.195377 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:30.195333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:30.297817 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:30.297759    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:31.230578 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:31.230546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:31.324932 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:31.324905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:31.533808 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:31.533751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:33.225579 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:33.225554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:39.235167 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:39.235137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:39.324364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:39.324329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:03:39.755231 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:39.755191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:39.870460 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:39.870421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:40.487070 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:40.486975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:46.734469 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:46.734438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:46.840964 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:46.840936    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:46.975768 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:46.975741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:47.474001 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:47.473964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:47.848878 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:47.848807    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:56.300111 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:56.300043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:56.440262 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:56.440232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:56.565393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:56.565335    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:57.982764 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:57.982730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:59.047172 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:59.047140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:03:59.930704 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:03:59.930662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:00.983291 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:00.983261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:01.771133 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:01.771104    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:01.866156 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:01.866131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:01.950811 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:01.950782    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:02.394221 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:02.394191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:02.496800 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:02.496775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:03.155898 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:03.155862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:04.091008 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:04.090978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:11.682597 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:11.682552    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:12.718297 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:12.718251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:13.380615 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:13.380578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:13.477421 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:13.477382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:14.160789 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:14.160756    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:14.903054 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:14.903019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:15.297514 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:15.297406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:16.020642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:16.020607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:16.979516 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:16.979483    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:17.604902 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:17.604848    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:04:18.807240 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:18.807196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:20.256068 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:20.256034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:21.365424 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:21.365389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:21.555666 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:21.555626    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:22.110203 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:22.110169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:22.306649 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:22.306625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:22.618221 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:22.618197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:23.041962 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:23.041918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:24.282236 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:24.282204    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:25.800941 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:25.800877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:27.062563 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:27.062503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:27.863006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:27.862956    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:27.962110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:27.962079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:28.037523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:28.037478    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:28.132409 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:28.132324    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:28.284622 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:28.284572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:28.374663 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:28.374613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:28.734708 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:28.734654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:29.382319 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:29.382288    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:30.088614 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:30.088566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:30.262378 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:30.262324    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:31.239020 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:31.238971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:32.690170 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:32.690136    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:33.600461 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:33.600418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:33.807151 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:33.807113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:34.558214 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.558177    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"]
Sep 18 19:04:34.561582 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.561565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.568484 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.568461    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-942883\"/\"e2e-conflict-a-af656e27-kserve-self-signed-certs\""
Sep 18 19:04:34.603944 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.603917    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"]
Sep 18 19:04:34.609072 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.609044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-dshm\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.609205 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.609101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-home\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.609205 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.609185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tls-certs\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.609336 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.609214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdm7k\" (UniqueName: \"kubernetes.io/projected/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-kube-api-access-kdm7k\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.609336 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.609269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tmp-dir\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.609336 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.609331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-model-cache\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.703186 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.703143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:34.710519 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.710493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-home\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.710597 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.710539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tls-certs\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.710597 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.710561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdm7k\" (UniqueName: \"kubernetes.io/projected/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-kube-api-access-kdm7k\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.710679 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.710633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tmp-dir\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.710731 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.710696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-model-cache\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.710772 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.710731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-dshm\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.710976 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.710957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-home\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.711039 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.711011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tmp-dir\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.711119 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.711101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-model-cache\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.713004 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.712988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-dshm\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.713187 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.713173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tls-certs\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.725227 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.725196    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdm7k\" (UniqueName: \"kubernetes.io/projected/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-kube-api-access-kdm7k\") pod \"e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:34.861121 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.861030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:34.869874 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:34.869853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:35.140774 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:35.140743    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"]
Sep 18 19:04:35.142356 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:04:35.142329    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87fbdaa7_eab6_4962_9c16_ff06b9812fd0.slice/crio-8816a85fb65ab1fd31f97b7927aef37a99fbb7b52982f809899fffbafdeddf40 WatchSource:0}: Error finding container 8816a85fb65ab1fd31f97b7927aef37a99fbb7b52982f809899fffbafdeddf40: Status 404 returned error can't find the container with id 8816a85fb65ab1fd31f97b7927aef37a99fbb7b52982f809899fffbafdeddf40
Sep 18 19:04:35.702458 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:35.702424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:35.848744 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:35.848664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn" event={"ID":"87fbdaa7-eab6-4962-9c16-ff06b9812fd0","Type":"ContainerStarted","Data":"255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66"}
Sep 18 19:04:35.848744 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:35.848700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn" event={"ID":"87fbdaa7-eab6-4962-9c16-ff06b9812fd0","Type":"ContainerStarted","Data":"8816a85fb65ab1fd31f97b7927aef37a99fbb7b52982f809899fffbafdeddf40"}
Sep 18 19:04:35.868820 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:35.868774    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn" podStartSLOduration=1.8687598109999999 podStartE2EDuration="1.868759811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:04:35.867164191 +0000 UTC m=+1199.659841149" watchObservedRunningTime="2026-09-18 19:04:35.868759811 +0000 UTC m=+1199.661436741"
Sep 18 19:04:36.433748 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:36.433706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:37.381656 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:37.381620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:40.569161 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:40.569124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:40.959346 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:40.959315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:41.858323 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:41.858286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:41.967519 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:41.967491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:42.067687 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:42.067659    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:42.368021 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:42.367969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:42.443477 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:42.443431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:42.563272 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:42.563233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:42.669015 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:42.668988    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:44.017921 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:44.017877    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:44.870760 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:44.870731    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:44.870994 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:44.870774    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:44.883042 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:44.883020    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:44.907727 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:44.907704    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:04:45.263818 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:45.263788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:46.649006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:46.648974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:47.824114 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:47.824072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:50.242500 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:50.242454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:50.366563 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:50.366536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:50.707342 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:50.707314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:51.129723 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:51.129642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:51.289472 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:51.289440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:51.486579 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:51.486547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:52.119378 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:52.119343    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:04:58.049444 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:58.049412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:58.176748 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:58.176713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:59.051472 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:59.051439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:59.663807 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:59.663770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:04:59.873516 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:04:59.873480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:00.030317 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:00.030212    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:00.129620 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:00.129576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:06.473440 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:06.473403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:06.572322 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:06.572292    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:08.195771 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:08.195734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:08.361393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:08.361361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:08.679450 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:08.679401    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:10.858215 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:10.858177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:11.703017 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:11.702987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:11.854334 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:11.854307    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:12.464972 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:12.464940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:12.628224 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:12.628174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:12.726946 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:12.726854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:13.401860 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:13.401804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:13.504702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:13.504675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:13.819758 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:13.819684    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:14.224948 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:14.224905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:14.600977 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:14.600873    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:14.705784 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:14.705754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:14.808426 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:14.808391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:15.358065 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:15.358027    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:05:15.891999 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:15.891969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:16.077839 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:16.077804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:16.289984 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:16.289904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:17.948289 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:17.948258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:18.066265 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:18.066237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:18.240327 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:18.240251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:18.573685 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:18.573602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:18.706930 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:18.706874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:29.958445 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:29.958418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:36.279433 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:36.279405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:36.401779 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:36.401749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:36.915707 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:36.915681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:38.108358 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:38.108329    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"]
Sep 18 19:05:38.141111 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:38.141085    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:41.363189 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:41.363162    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:41.628556 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:41.628490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:41.978954 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:41.978925    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:42.388524 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:42.388450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:42.472854 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:42.472827    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:42.676085 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:42.676064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:43.322572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:43.322548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:05:43.560101 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:43.560074    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:44.738898 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:44.738847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:45.228854 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:45.228821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:45.417704 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:45.417671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:46.179394 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:46.179370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:48.094695 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:48.094664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:49.877655 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:49.877619    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:50.026044 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:50.026017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:50.185591 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:50.185566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:50.798057 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:50.798033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:51.051861 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.051795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:51.115040 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.115008    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"]
Sep 18 19:05:51.118381 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.118365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:51.127369 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:05:51.127343    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="8515879d-6208-411e-b787-ae39b7c5e9eb" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:51.127463 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:05:51.127391    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-shared-b-w2-4805a7-dockercfg-xr8zt\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4805a7-dockercfg-xr8zt\"" type="*v1.Secret"
Sep 18 19:05:51.127531 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:05:51.127507    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-shared-b-w2-4805a7\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4805a7\"" type="*v1.ConfigMap"
Sep 18 19:05:51.137402 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.137377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlsx\" (UniqueName: \"kubernetes.io/projected/8515879d-6208-411e-b787-ae39b7c5e9eb-kube-api-access-fwlsx\") pod \"payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:51.137452 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.137441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515879d-6208-411e-b787-ae39b7c5e9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:51.150457 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.150432    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"]
Sep 18 19:05:51.237868 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.237844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlsx\" (UniqueName: \"kubernetes.io/projected/8515879d-6208-411e-b787-ae39b7c5e9eb-kube-api-access-fwlsx\") pod \"payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:51.237983 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.237920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515879d-6208-411e-b787-ae39b7c5e9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:51.250576 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.250547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlsx\" (UniqueName: \"kubernetes.io/projected/8515879d-6208-411e-b787-ae39b7c5e9eb-kube-api-access-fwlsx\") pod \"payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:51.574149 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.574124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:51.979766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.979741    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4805a7\""
Sep 18 19:05:51.989421 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:51.989401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515879d-6208-411e-b787-ae39b7c5e9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:52.583063 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:52.583040    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4805a7-dockercfg-xr8zt\""
Sep 18 19:05:52.588972 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:52.588952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:52.642029 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:52.642008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:52.875314 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:05:52.875274    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8515879d_6208_411e_b787_ae39b7c5e9eb.slice/crio-a975c8bda71298f5b6d1ea87218cabc22ccfc9967a550d9c91d18f884ef0344e WatchSource:0}: Error finding container a975c8bda71298f5b6d1ea87218cabc22ccfc9967a550d9c91d18f884ef0344e: Status 404 returned error can't find the container with id a975c8bda71298f5b6d1ea87218cabc22ccfc9967a550d9c91d18f884ef0344e
Sep 18 19:05:52.876406 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:52.876384    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"]
Sep 18 19:05:53.127218 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.127135    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c" podUID="227e5808-19aa-4e62-a215-95ab04c056c1" containerName="main" containerID="cri-o://847d6b70a9028238df5cc7ffe3e4e9a136ebfa438580e61f04256cc82a874a0e" gracePeriod=45
Sep 18 19:05:53.250514 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.250459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7" event={"ID":"8515879d-6208-411e-b787-ae39b7c5e9eb","Type":"ContainerStarted","Data":"da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e"}
Sep 18 19:05:53.250640 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.250535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7" event={"ID":"8515879d-6208-411e-b787-ae39b7c5e9eb","Type":"ContainerStarted","Data":"a975c8bda71298f5b6d1ea87218cabc22ccfc9967a550d9c91d18f884ef0344e"}
Sep 18 19:05:53.250640 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.250566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:05:53.252305 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.252274    2582 generic.go:356] "Generic (PLEG): container finished" podID="227e5808-19aa-4e62-a215-95ab04c056c1" containerID="847d6b70a9028238df5cc7ffe3e4e9a136ebfa438580e61f04256cc82a874a0e" exitCode=0
Sep 18 19:05:53.252415 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.252353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c" event={"ID":"227e5808-19aa-4e62-a215-95ab04c056c1","Type":"ContainerDied","Data":"847d6b70a9028238df5cc7ffe3e4e9a136ebfa438580e61f04256cc82a874a0e"}
Sep 18 19:05:53.358003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.357978    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:05:53.453963 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.453916    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7" podStartSLOduration=2.453900673 podStartE2EDuration="2.453900673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:05:53.283837408 +0000 UTC m=+1277.076514356" watchObservedRunningTime="2026-09-18 19:05:53.453900673 +0000 UTC m=+1277.246577596"
Sep 18 19:05:53.455336 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.455307    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227e5808-19aa-4e62-a215-95ab04c056c1-tls-certs\") pod \"227e5808-19aa-4e62-a215-95ab04c056c1\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") "
Sep 18 19:05:53.455475 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.455400    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-tmp-dir\") pod \"227e5808-19aa-4e62-a215-95ab04c056c1\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") "
Sep 18 19:05:53.455475 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.455446    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-model-cache\") pod \"227e5808-19aa-4e62-a215-95ab04c056c1\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") "
Sep 18 19:05:53.455601 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.455482    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sb95\" (UniqueName: \"kubernetes.io/projected/227e5808-19aa-4e62-a215-95ab04c056c1-kube-api-access-4sb95\") pod \"227e5808-19aa-4e62-a215-95ab04c056c1\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") "
Sep 18 19:05:53.455601 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.455523    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-home\") pod \"227e5808-19aa-4e62-a215-95ab04c056c1\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") "
Sep 18 19:05:53.455713 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.455655    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "227e5808-19aa-4e62-a215-95ab04c056c1" (UID: "227e5808-19aa-4e62-a215-95ab04c056c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:53.455835 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.455815    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "227e5808-19aa-4e62-a215-95ab04c056c1" (UID: "227e5808-19aa-4e62-a215-95ab04c056c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:53.456796 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.456207    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-home" (OuterVolumeSpecName: "home") pod "227e5808-19aa-4e62-a215-95ab04c056c1" (UID: "227e5808-19aa-4e62-a215-95ab04c056c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:53.456796 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.456305    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-dshm\") pod \"227e5808-19aa-4e62-a215-95ab04c056c1\" (UID: \"227e5808-19aa-4e62-a215-95ab04c056c1\") "
Sep 18 19:05:53.457319 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.457234    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:05:53.457319 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.457273    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:05:53.457319 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.457285    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:05:53.459330 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.459300    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-dshm" (OuterVolumeSpecName: "dshm") pod "227e5808-19aa-4e62-a215-95ab04c056c1" (UID: "227e5808-19aa-4e62-a215-95ab04c056c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:05:53.468204 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.468182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227e5808-19aa-4e62-a215-95ab04c056c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "227e5808-19aa-4e62-a215-95ab04c056c1" (UID: "227e5808-19aa-4e62-a215-95ab04c056c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:05:53.469829 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.469805    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227e5808-19aa-4e62-a215-95ab04c056c1-kube-api-access-4sb95" (OuterVolumeSpecName: "kube-api-access-4sb95") pod "227e5808-19aa-4e62-a215-95ab04c056c1" (UID: "227e5808-19aa-4e62-a215-95ab04c056c1"). InnerVolumeSpecName "kube-api-access-4sb95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:05:53.558291 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.558263    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/227e5808-19aa-4e62-a215-95ab04c056c1-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:05:53.558291 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.558284    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/227e5808-19aa-4e62-a215-95ab04c056c1-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:05:53.558291 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:53.558293    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sb95\" (UniqueName: \"kubernetes.io/projected/227e5808-19aa-4e62-a215-95ab04c056c1-kube-api-access-4sb95\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:05:54.258449 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:54.258428    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"
Sep 18 19:05:54.258896 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:54.258457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c" event={"ID":"227e5808-19aa-4e62-a215-95ab04c056c1","Type":"ContainerDied","Data":"025a25182d3e8c3603fb639227bbc31bf49cef537382aad084f8e9ae58e95f6b"}
Sep 18 19:05:54.258896 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:54.258499    2582 scope.go:117] "RemoveContainer" containerID="847d6b70a9028238df5cc7ffe3e4e9a136ebfa438580e61f04256cc82a874a0e"
Sep 18 19:05:54.291413 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:54.291386    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"]
Sep 18 19:05:54.300803 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:54.300782    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-15c3ed/test-model-15c3ed-kserve-7cdd457899-x4b9c"]
Sep 18 19:05:54.334787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:54.334762    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="227e5808-19aa-4e62-a215-95ab04c056c1" path="/var/lib/kubelet/pods/227e5808-19aa-4e62-a215-95ab04c056c1/volumes"
Sep 18 19:05:54.394420 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:54.394398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:05:55.602253 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:05:55.602228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:04.260352 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:04.260325    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:06:05.802582 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:05.802539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:06:09.577258 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.577225    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"]
Sep 18 19:06:09.577619 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.577585    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227e5808-19aa-4e62-a215-95ab04c056c1" containerName="main"
Sep 18 19:06:09.577619 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.577595    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="227e5808-19aa-4e62-a215-95ab04c056c1" containerName="main"
Sep 18 19:06:09.577689 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.577664    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="227e5808-19aa-4e62-a215-95ab04c056c1" containerName="main"
Sep 18 19:06:09.580671 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.580656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.593070 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.593051    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-942883\"/\"e2e-conflict-b-af656e27-kserve-self-signed-certs\""
Sep 18 19:06:09.611384 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.611363    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"]
Sep 18 19:06:09.636622 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.636600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:09.683521 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.683496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-dshm\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.683602 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.683538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-home\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.683602 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.683590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-tmp-dir\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.683679 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.683615    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj28z\" (UniqueName: \"kubernetes.io/projected/650e2b95-7549-4a90-a1df-06257f3a8a2c-kube-api-access-qj28z\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.683679 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.683661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-model-cache\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.683743 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.683695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650e2b95-7549-4a90-a1df-06257f3a8a2c-tls-certs\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784369 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-dshm\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-home\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-tmp-dir\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj28z\" (UniqueName: \"kubernetes.io/projected/650e2b95-7549-4a90-a1df-06257f3a8a2c-kube-api-access-qj28z\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784617 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-model-cache\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784671 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650e2b95-7549-4a90-a1df-06257f3a8a2c-tls-certs\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784772 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-home\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784819 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-tmp-dir\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.784855 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.784843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-model-cache\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.786934 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.786906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-dshm\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.787129 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.787113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650e2b95-7549-4a90-a1df-06257f3a8a2c-tls-certs\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.800771 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.800744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj28z\" (UniqueName: \"kubernetes.io/projected/650e2b95-7549-4a90-a1df-06257f3a8a2c-kube-api-access-qj28z\") pod \"e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") " pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.888873 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.888822    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:09.977466 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:09.977439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:10.191180 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:06:10.191152    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod650e2b95_7549_4a90_a1df_06257f3a8a2c.slice/crio-1eceff5cd9e2ac4f7c17fb8eb502df4d6ae795ad48cd4e82ae8ce091fa18de3d WatchSource:0}: Error finding container 1eceff5cd9e2ac4f7c17fb8eb502df4d6ae795ad48cd4e82ae8ce091fa18de3d: Status 404 returned error can't find the container with id 1eceff5cd9e2ac4f7c17fb8eb502df4d6ae795ad48cd4e82ae8ce091fa18de3d
Sep 18 19:06:10.192507 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:10.192481    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"]
Sep 18 19:06:10.343798 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:10.343757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86" event={"ID":"650e2b95-7549-4a90-a1df-06257f3a8a2c","Type":"ContainerStarted","Data":"907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7"}
Sep 18 19:06:10.343998 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:10.343808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86" event={"ID":"650e2b95-7549-4a90-a1df-06257f3a8a2c","Type":"ContainerStarted","Data":"1eceff5cd9e2ac4f7c17fb8eb502df4d6ae795ad48cd4e82ae8ce091fa18de3d"}
Sep 18 19:06:10.375318 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:10.375275    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86" podStartSLOduration=1.375262805 podStartE2EDuration="1.375262805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:06:10.375142873 +0000 UTC m=+1294.167819800" watchObservedRunningTime="2026-09-18 19:06:10.375262805 +0000 UTC m=+1294.167939734"
Sep 18 19:06:11.382307 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:11.382277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:12.783541 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:12.783510    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:13.937818 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:13.937785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:15.076752 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:15.076721    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:16.174481 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:16.174453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:17.412337 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:17.412306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:18.797921 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:18.797867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:19.889103 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:19.889077    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:19.889475 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:19.889313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:19.901329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:19.901301    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:19.990992 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:19.990962    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:20.404878 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:20.404853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:06:20.516306 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:20.516277    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:21.300093 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.300059    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:21.391395 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.391359    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-5703c9"]
Sep 18 19:06:21.395130 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.395105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:21.397594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.397572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wgm9b\""
Sep 18 19:06:21.408187 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.408164    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-5703c9"]
Sep 18 19:06:21.484555 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.484521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5h8\" (UniqueName: \"kubernetes.io/projected/de4ddad3-64f4-4882-a1a5-cece96e10631-kube-api-access-4r5h8\") pod \"test-curl-7681-5703c9\" (UID: \"de4ddad3-64f4-4882-a1a5-cece96e10631\") " pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:21.585786 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.585702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5h8\" (UniqueName: \"kubernetes.io/projected/de4ddad3-64f4-4882-a1a5-cece96e10631-kube-api-access-4r5h8\") pod \"test-curl-7681-5703c9\" (UID: \"de4ddad3-64f4-4882-a1a5-cece96e10631\") " pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:21.602115 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.602082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5h8\" (UniqueName: \"kubernetes.io/projected/de4ddad3-64f4-4882-a1a5-cece96e10631-kube-api-access-4r5h8\") pod \"test-curl-7681-5703c9\" (UID: \"de4ddad3-64f4-4882-a1a5-cece96e10631\") " pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:21.705583 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.705547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:21.985298 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:21.985271    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-5703c9"]
Sep 18 19:06:21.986435 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:06:21.986396    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde4ddad3_64f4_4882_a1a5_cece96e10631.slice/crio-55853a06e18ce02fb5ac38d72545183498899c0d418f34f46268f165f1f2c91e WatchSource:0}: Error finding container 55853a06e18ce02fb5ac38d72545183498899c0d418f34f46268f165f1f2c91e: Status 404 returned error can't find the container with id 55853a06e18ce02fb5ac38d72545183498899c0d418f34f46268f165f1f2c91e
Sep 18 19:06:22.404401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:22.404372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-5703c9" event={"ID":"de4ddad3-64f4-4882-a1a5-cece96e10631","Type":"ContainerStarted","Data":"b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e"}
Sep 18 19:06:22.404845 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:22.404424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-5703c9" event={"ID":"de4ddad3-64f4-4882-a1a5-cece96e10631","Type":"ContainerStarted","Data":"55853a06e18ce02fb5ac38d72545183498899c0d418f34f46268f165f1f2c91e"}
Sep 18 19:06:22.428987 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:22.428942    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-5703c9" podStartSLOduration=1.4289265119999999 podStartE2EDuration="1.428926512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:06:22.427233299 +0000 UTC m=+1306.219910229" watchObservedRunningTime="2026-09-18 19:06:22.428926512 +0000 UTC m=+1306.221603441"
Sep 18 19:06:22.564681 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:22.564654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:23.282159 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:23.282093    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-5703c9"]
Sep 18 19:06:23.290737 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:23.290710    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-5703c9"]
Sep 18 19:06:23.409569 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:06:23.409542    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-5703c9" container="test-curl-7681-5703c9"
Sep 18 19:06:24.177374 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:24.177326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:24.414570 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:24.414518    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-5703c9" podUID="de4ddad3-64f4-4882-a1a5-cece96e10631" containerName="test-curl-7681-5703c9" containerID="cri-o://b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e" gracePeriod=2
Sep 18 19:06:25.279923 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:25.279875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:26.177800 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:26.177750    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:26.593770 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:26.593743    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:26.726766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:26.726694    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r5h8\" (UniqueName: \"kubernetes.io/projected/de4ddad3-64f4-4882-a1a5-cece96e10631-kube-api-access-4r5h8\") pod \"de4ddad3-64f4-4882-a1a5-cece96e10631\" (UID: \"de4ddad3-64f4-4882-a1a5-cece96e10631\") "
Sep 18 19:06:26.728995 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:26.728970    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4ddad3-64f4-4882-a1a5-cece96e10631-kube-api-access-4r5h8" (OuterVolumeSpecName: "kube-api-access-4r5h8") pod "de4ddad3-64f4-4882-a1a5-cece96e10631" (UID: "de4ddad3-64f4-4882-a1a5-cece96e10631"). InnerVolumeSpecName "kube-api-access-4r5h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:06:26.828252 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:26.828226    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5h8\" (UniqueName: \"kubernetes.io/projected/de4ddad3-64f4-4882-a1a5-cece96e10631-kube-api-access-4r5h8\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:06:27.048073 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.047992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:27.215256 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.215222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:27.300170 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.300093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:27.404074 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.404041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:27.431838 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.431811    2582 generic.go:356] "Generic (PLEG): container finished" podID="de4ddad3-64f4-4882-a1a5-cece96e10631" containerID="b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e" exitCode=137
Sep 18 19:06:27.432015 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.431852    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:27.432015 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.431928    2582 scope.go:117] "RemoveContainer" containerID="b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e"
Sep 18 19:06:27.442021 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.442000    2582 scope.go:117] "RemoveContainer" containerID="b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e"
Sep 18 19:06:27.442312 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:06:27.442293    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e\": container with ID starting with b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e not found: ID does not exist" containerID="b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e"
Sep 18 19:06:27.442391 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.442323    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e"} err="failed to get container status \"b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e\": rpc error: code = NotFound desc = could not find container \"b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e\": container with ID starting with b36799b9cdafdb21dc6fdd2bd10f01e33492b76e11de04f63776b4612f7c270e not found: ID does not exist"
Sep 18 19:06:27.443329 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:06:27.443301    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-5703c9\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="de4ddad3-64f4-4882-a1a5-cece96e10631" pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:27.539296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.539272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:27.825989 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.825953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:27.985796 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:27.985755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:28.333696 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:06:28.333660    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-5703c9\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="de4ddad3-64f4-4882-a1a5-cece96e10631" pod="opendatahub/test-curl-7681-5703c9"
Sep 18 19:06:28.335302 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:28.335278    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4ddad3-64f4-4882-a1a5-cece96e10631" path="/var/lib/kubelet/pods/de4ddad3-64f4-4882-a1a5-cece96e10631/volumes"
Sep 18 19:06:29.072419 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:29.072390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:30.227166 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:30.227127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:31.335109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:31.335058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:32.109469 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:32.109437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:32.294197 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:32.294166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:32.548433 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:32.548403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:32.937657 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:32.937621    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:33.282922 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.282795    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"]
Sep 18 19:06:33.283261 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.283243    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4ddad3-64f4-4882-a1a5-cece96e10631" containerName="test-curl-7681-5703c9"
Sep 18 19:06:33.283261 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.283262    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4ddad3-64f4-4882-a1a5-cece96e10631" containerName="test-curl-7681-5703c9"
Sep 18 19:06:33.283411 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.283345    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4ddad3-64f4-4882-a1a5-cece96e10631" containerName="test-curl-7681-5703c9"
Sep 18 19:06:33.287836 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.287815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.290270 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.290249    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9674d9\"/\"kube-root-ca.crt\""
Sep 18 19:06:33.290991 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.290966    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9674d9\"/\"default-dockercfg-qjpkw\""
Sep 18 19:06:33.291092 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.291032    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9674d9\"/\"noten-model-9674d9-kserve-self-signed-certs\""
Sep 18 19:06:33.291160 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.291092    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9674d9\"/\"openshift-service-ca.crt\""
Sep 18 19:06:33.310397 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.310374    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"]
Sep 18 19:06:33.389179 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.389138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-home\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.389296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.389189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-model-cache\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.389296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.389224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tls-certs\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.389296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.389276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tmp-dir\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.389446 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.389305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcv2\" (UniqueName: \"kubernetes.io/projected/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-kube-api-access-kqcv2\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.389446 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.389381    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-dshm\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.389918 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.389863    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:33.490719 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.490693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-home\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.490918 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.490836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-model-cache\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.491006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.490915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tls-certs\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.491006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.491001    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tmp-dir\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.491093 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.491046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcv2\" (UniqueName: \"kubernetes.io/projected/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-kube-api-access-kqcv2\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.491093 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.491076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-home\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.491093 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.491087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-dshm\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.491234 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.491184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-model-cache\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.491291 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.491251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tmp-dir\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.493430 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.493407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-dshm\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.493676 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.493659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tls-certs\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.506852 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.506832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcv2\" (UniqueName: \"kubernetes.io/projected/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-kube-api-access-kqcv2\") pod \"noten-model-9674d9-kserve-59b5f7d7ff-mdz5k\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") " pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.597731 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.597643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:33.888044 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:33.888013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"]
Sep 18 19:06:33.889397 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:06:33.889371    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc1fb6dd_6a5c_47d6_a7c0_dc9fd377e016.slice/crio-102a673f0e05dd08b37f495526cf87a115dd9e7d0b1568cd350d26841a247b2e WatchSource:0}: Error finding container 102a673f0e05dd08b37f495526cf87a115dd9e7d0b1568cd350d26841a247b2e: Status 404 returned error can't find the container with id 102a673f0e05dd08b37f495526cf87a115dd9e7d0b1568cd350d26841a247b2e
Sep 18 19:06:34.103660 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:34.103625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:34.472627 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:34.472588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k" event={"ID":"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016","Type":"ContainerStarted","Data":"6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146"}
Sep 18 19:06:34.472808 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:34.472642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k" event={"ID":"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016","Type":"ContainerStarted","Data":"102a673f0e05dd08b37f495526cf87a115dd9e7d0b1568cd350d26841a247b2e"}
Sep 18 19:06:34.523438 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:34.523386    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k" podStartSLOduration=1.523369601 podStartE2EDuration="1.523369601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:06:34.521771555 +0000 UTC m=+1318.314448503" watchObservedRunningTime="2026-09-18 19:06:34.523369601 +0000 UTC m=+1318.316046532"
Sep 18 19:06:35.445768 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:35.445731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:36.186127 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:36.186095    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:36.302766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:36.302728    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:36.841090 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:36.841053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:37.146183 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:37.146149    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:06:38.037523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:38.037494    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:38.332477 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:38.332387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:38.416623 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:38.416591    2582 scope.go:117] "RemoveContainer" containerID="9efe2c23536bcb837cf30f4ffe1555ce7a2ed756bd1516ab376d353fdd6c5c00"
Sep 18 19:06:38.495094 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:38.495065    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:38.769202 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:38.769169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:39.704993 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:39.704963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:40.880759 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:40.880725    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:41.753510 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:41.753474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:42.782542 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:42.782512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:42.884533 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:42.884492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:42.983859 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:42.983828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:43.442146 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.442112    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:43.543764 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.543737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:43.598374 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.598337    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:43.598538 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.598396    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:43.611511 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.611484    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:43.646377 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.646344    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"]
Sep 18 19:06:43.652330 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.652295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:43.656441 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.656415    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4ac9ac7d-dockercfg-q9drx\""
Sep 18 19:06:43.656581 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.656502    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4ac9ac7d\""
Sep 18 19:06:43.663290 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.663268    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"]
Sep 18 19:06:43.677310 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.677271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36a823c2-2227-4fcb-82a2-881ee1672aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:43.677442 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.677333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkmc\" (UniqueName: \"kubernetes.io/projected/36a823c2-2227-4fcb-82a2-881ee1672aec-kube-api-access-mgkmc\") pod \"payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:43.778229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.778131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkmc\" (UniqueName: \"kubernetes.io/projected/36a823c2-2227-4fcb-82a2-881ee1672aec-kube-api-access-mgkmc\") pod \"payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:43.778410 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.778235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36a823c2-2227-4fcb-82a2-881ee1672aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:43.778958 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.778938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36a823c2-2227-4fcb-82a2-881ee1672aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:43.787642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.787621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkmc\" (UniqueName: \"kubernetes.io/projected/36a823c2-2227-4fcb-82a2-881ee1672aec-kube-api-access-mgkmc\") pod \"payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:43.955355 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.955303    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:43.965214 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:43.965184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:44.256507 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:44.256473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"]
Sep 18 19:06:44.260657 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:06:44.260623    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a823c2_2227_4fcb_82a2_881ee1672aec.slice/crio-1bc0180943d0207bf063395758e0789c98c3d41df48d73dc302613c101348a02 WatchSource:0}: Error finding container 1bc0180943d0207bf063395758e0789c98c3d41df48d73dc302613c101348a02: Status 404 returned error can't find the container with id 1bc0180943d0207bf063395758e0789c98c3d41df48d73dc302613c101348a02
Sep 18 19:06:44.526602 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:44.526513    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t" event={"ID":"36a823c2-2227-4fcb-82a2-881ee1672aec","Type":"ContainerStarted","Data":"945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994"}
Sep 18 19:06:44.526786 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:44.526651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:44.526786 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:44.526692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t" event={"ID":"36a823c2-2227-4fcb-82a2-881ee1672aec","Type":"ContainerStarted","Data":"1bc0180943d0207bf063395758e0789c98c3d41df48d73dc302613c101348a02"}
Sep 18 19:06:44.538124 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:44.538103    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:06:44.560064 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:44.560010    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t" podStartSLOduration=1.559991615 podStartE2EDuration="1.559991615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:06:44.557664246 +0000 UTC m=+1328.350341199" watchObservedRunningTime="2026-09-18 19:06:44.559991615 +0000 UTC m=+1328.352668551"
Sep 18 19:06:45.122408 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:45.122366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:46.205077 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:46.205042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:47.147083 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:47.147054    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:47.459628 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:47.459548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:48.357937 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:48.357902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:49.691434 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:49.691359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:49.757172 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:49.757138    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"]
Sep 18 19:06:49.757437 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:49.757409    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr" podUID="bfa30389-314c-4369-a0ac-84cbca9a0e5f" containerName="payload-processing" containerID="cri-o://790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0" gracePeriod=30
Sep 18 19:06:49.759596 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:49.759562    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"]
Sep 18 19:06:49.759751 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:49.759729    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw" podUID="709e318c-09aa-465d-a108-9cb6833371ea" containerName="payload-pre-processing" containerID="cri-o://675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85" gracePeriod=30
Sep 18 19:06:49.794024 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:49.794000    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:49.895465 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:49.895398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:50.015047 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.015021    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 19:06:50.040627 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.040606    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 19:06:50.045069 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.045044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:50.141039 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.141006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbm6z\" (UniqueName: \"kubernetes.io/projected/709e318c-09aa-465d-a108-9cb6833371ea-kube-api-access-vbm6z\") pod \"709e318c-09aa-465d-a108-9cb6833371ea\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") "
Sep 18 19:06:50.141229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.141050    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa30389-314c-4369-a0ac-84cbca9a0e5f-plugins-config-volume\") pod \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") "
Sep 18 19:06:50.141229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.141161    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709e318c-09aa-465d-a108-9cb6833371ea-plugins-config-volume\") pod \"709e318c-09aa-465d-a108-9cb6833371ea\" (UID: \"709e318c-09aa-465d-a108-9cb6833371ea\") "
Sep 18 19:06:50.141229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.141198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5szf5\" (UniqueName: \"kubernetes.io/projected/bfa30389-314c-4369-a0ac-84cbca9a0e5f-kube-api-access-5szf5\") pod \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\" (UID: \"bfa30389-314c-4369-a0ac-84cbca9a0e5f\") "
Sep 18 19:06:50.141475 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.141449    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa30389-314c-4369-a0ac-84cbca9a0e5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfa30389-314c-4369-a0ac-84cbca9a0e5f" (UID: "bfa30389-314c-4369-a0ac-84cbca9a0e5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:06:50.141572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.141551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709e318c-09aa-465d-a108-9cb6833371ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "709e318c-09aa-465d-a108-9cb6833371ea" (UID: "709e318c-09aa-465d-a108-9cb6833371ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:06:50.143390 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.143368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709e318c-09aa-465d-a108-9cb6833371ea-kube-api-access-vbm6z" (OuterVolumeSpecName: "kube-api-access-vbm6z") pod "709e318c-09aa-465d-a108-9cb6833371ea" (UID: "709e318c-09aa-465d-a108-9cb6833371ea"). InnerVolumeSpecName "kube-api-access-vbm6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:06:50.143479 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.143396    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa30389-314c-4369-a0ac-84cbca9a0e5f-kube-api-access-5szf5" (OuterVolumeSpecName: "kube-api-access-5szf5") pod "bfa30389-314c-4369-a0ac-84cbca9a0e5f" (UID: "bfa30389-314c-4369-a0ac-84cbca9a0e5f"). InnerVolumeSpecName "kube-api-access-5szf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:06:50.242370 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.242298    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/709e318c-09aa-465d-a108-9cb6833371ea-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:06:50.242370 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.242322    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5szf5\" (UniqueName: \"kubernetes.io/projected/bfa30389-314c-4369-a0ac-84cbca9a0e5f-kube-api-access-5szf5\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:06:50.242370 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.242332    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbm6z\" (UniqueName: \"kubernetes.io/projected/709e318c-09aa-465d-a108-9cb6833371ea-kube-api-access-vbm6z\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:06:50.242370 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.242340    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa30389-314c-4369-a0ac-84cbca9a0e5f-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:06:50.559867 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.559789    2582 generic.go:356] "Generic (PLEG): container finished" podID="bfa30389-314c-4369-a0ac-84cbca9a0e5f" containerID="790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0" exitCode=0
Sep 18 19:06:50.559867 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.559842    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"
Sep 18 19:06:50.560092 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.559900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr" event={"ID":"bfa30389-314c-4369-a0ac-84cbca9a0e5f","Type":"ContainerDied","Data":"790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0"}
Sep 18 19:06:50.560092 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.559959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr" event={"ID":"bfa30389-314c-4369-a0ac-84cbca9a0e5f","Type":"ContainerDied","Data":"b7bb983212a7582352f547ab69af4eff0f44600001081304b26a973f19b8fdc9"}
Sep 18 19:06:50.560092 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.559986    2582 scope.go:117] "RemoveContainer" containerID="790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0"
Sep 18 19:06:50.561414 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.561391    2582 generic.go:356] "Generic (PLEG): container finished" podID="709e318c-09aa-465d-a108-9cb6833371ea" containerID="675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85" exitCode=0
Sep 18 19:06:50.561515 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.561431    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"
Sep 18 19:06:50.561515 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.561433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw" event={"ID":"709e318c-09aa-465d-a108-9cb6833371ea","Type":"ContainerDied","Data":"675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85"}
Sep 18 19:06:50.561667 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.561543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw" event={"ID":"709e318c-09aa-465d-a108-9cb6833371ea","Type":"ContainerDied","Data":"ee49b9ab4de48717d00220c19634a707c57ed844c8accf028b9eb3b8a00ab779"}
Sep 18 19:06:50.569211 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.569185    2582 scope.go:117] "RemoveContainer" containerID="790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0"
Sep 18 19:06:50.569494 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:06:50.569478    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0\": container with ID starting with 790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0 not found: ID does not exist" containerID="790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0"
Sep 18 19:06:50.569540 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.569502    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0"} err="failed to get container status \"790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0\": rpc error: code = NotFound desc = could not find container \"790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0\": container with ID starting with 790d26bdec14bfca0b4f968cf98fc54795a01a35c93782b7b6d8fb524523a9f0 not found: ID does not exist"
Sep 18 19:06:50.569540 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.569521    2582 scope.go:117] "RemoveContainer" containerID="675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85"
Sep 18 19:06:50.577615 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.577599    2582 scope.go:117] "RemoveContainer" containerID="675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85"
Sep 18 19:06:50.577894 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:06:50.577860    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85\": container with ID starting with 675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85 not found: ID does not exist" containerID="675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85"
Sep 18 19:06:50.577968 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.577911    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85"} err="failed to get container status \"675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85\": rpc error: code = NotFound desc = could not find container \"675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85\": container with ID starting with 675ed8d010f732aad2483dff6f053741827e3fad89f48e9dfaf62fa56e40af85 not found: ID does not exist"
Sep 18 19:06:50.601383 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.601349    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"]
Sep 18 19:06:50.607650 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.607625    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3a81e5-5c44bb67b9-m4vwr"]
Sep 18 19:06:50.652808 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.652776    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"]
Sep 18 19:06:50.669554 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.669521    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3a81e5-974bb8998-5nkkw"]
Sep 18 19:06:50.749011 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.748964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:50.943167 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:50.943143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:51.828063 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:51.828037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:52.335819 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:52.335783    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="709e318c-09aa-465d-a108-9cb6833371ea" path="/var/lib/kubelet/pods/709e318c-09aa-465d-a108-9cb6833371ea/volumes"
Sep 18 19:06:52.336214 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:52.336196    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa30389-314c-4369-a0ac-84cbca9a0e5f" path="/var/lib/kubelet/pods/bfa30389-314c-4369-a0ac-84cbca9a0e5f/volumes"
Sep 18 19:06:53.382701 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:53.382667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:54.669590 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:54.669558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:55.533057 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:55.533020    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:06:55.738964 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:55.738931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:55.976206 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:55.976165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:56.245379 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:56.245284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:57.440316 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:57.440283    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:06:59.162609 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:06:59.162573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:00.970470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:00.970437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:02.409297 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:02.409260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:03.745267 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:03.745225    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:04.878205 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:04.878165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:05.181445 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:05.181413    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:05.331128 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:05.331094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:09.500066 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:09.500029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:10.257086 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:10.257042    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"]
Sep 18 19:07:10.547488 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:10.547376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:12.486622 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:12.486588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:21.531011 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:21.530964    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:07:25.277365 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.277323    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k" podUID="cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" containerName="main" containerID="cri-o://6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146" gracePeriod=45
Sep 18 19:07:25.534482 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.534413    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:07:25.589458 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-home\") pod \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") "
Sep 18 19:07:25.589662 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-model-cache\") pod \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") "
Sep 18 19:07:25.589662 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqcv2\" (UniqueName: \"kubernetes.io/projected/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-kube-api-access-kqcv2\") pod \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") "
Sep 18 19:07:25.589662 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tls-certs\") pod \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") "
Sep 18 19:07:25.589662 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-dshm\") pod \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") "
Sep 18 19:07:25.589969 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tmp-dir\") pod \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\" (UID: \"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016\") "
Sep 18 19:07:25.589969 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589768    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-home" (OuterVolumeSpecName: "home") pod "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" (UID: "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:07:25.589969 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.589814    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" (UID: "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:07:25.590116 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.590021    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" (UID: "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:07:25.590116 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.590105    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:07:25.590203 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.590125    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:07:25.592396 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.592368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-kube-api-access-kqcv2" (OuterVolumeSpecName: "kube-api-access-kqcv2") pod "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" (UID: "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016"). InnerVolumeSpecName "kube-api-access-kqcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:07:25.592528 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.592472    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-dshm" (OuterVolumeSpecName: "dshm") pod "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" (UID: "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:07:25.592528 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.592507    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" (UID: "cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:07:25.691631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.691585    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcv2\" (UniqueName: \"kubernetes.io/projected/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-kube-api-access-kqcv2\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:07:25.691631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.691624    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:07:25.691631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.691634    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:07:25.691631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.691641    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:07:25.757140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.757098    2582 generic.go:356] "Generic (PLEG): container finished" podID="cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" containerID="6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146" exitCode=0
Sep 18 19:07:25.757331 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.757149    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"
Sep 18 19:07:25.757331 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.757189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k" event={"ID":"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016","Type":"ContainerDied","Data":"6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146"}
Sep 18 19:07:25.757331 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.757235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k" event={"ID":"cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016","Type":"ContainerDied","Data":"102a673f0e05dd08b37f495526cf87a115dd9e7d0b1568cd350d26841a247b2e"}
Sep 18 19:07:25.757331 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.757254    2582 scope.go:117] "RemoveContainer" containerID="6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146"
Sep 18 19:07:25.774137 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.774111    2582 scope.go:117] "RemoveContainer" containerID="6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146"
Sep 18 19:07:25.774463 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:07:25.774439    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146\": container with ID starting with 6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146 not found: ID does not exist" containerID="6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146"
Sep 18 19:07:25.774546 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.774471    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146"} err="failed to get container status \"6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146\": rpc error: code = NotFound desc = could not find container \"6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146\": container with ID starting with 6731cc9e300cb83605aaa0419cb9d47151b581d21cc0cdb14610f209b848f146 not found: ID does not exist"
Sep 18 19:07:25.796260 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.796165    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"]
Sep 18 19:07:25.805578 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:25.805547    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-9674d9/noten-model-9674d9-kserve-59b5f7d7ff-mdz5k"]
Sep 18 19:07:26.336341 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:26.336303    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" path="/var/lib/kubelet/pods/cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016/volumes"
Sep 18 19:07:33.925844 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:33.925810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:34.205123 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:34.205023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:34.812421 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:34.812383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:35.153995 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:35.153946    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:35.293589 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:35.293544    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:37.732634 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:37.732588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:38.403854 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:38.403818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:38.575865 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:38.575802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:38.738656 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:38.738521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:40.151177 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:40.151108    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:41.209080 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:41.209018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:41.990592 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:41.990542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:42.726472 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:42.726411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:43.718491 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.718453    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"]
Sep 18 19:07:43.718929 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.718899    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa30389-314c-4369-a0ac-84cbca9a0e5f" containerName="payload-processing"
Sep 18 19:07:43.718929 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.718921    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa30389-314c-4369-a0ac-84cbca9a0e5f" containerName="payload-processing"
Sep 18 19:07:43.719140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.718957    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="709e318c-09aa-465d-a108-9cb6833371ea" containerName="payload-pre-processing"
Sep 18 19:07:43.719140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.718965    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="709e318c-09aa-465d-a108-9cb6833371ea" containerName="payload-pre-processing"
Sep 18 19:07:43.719140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.718991    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" containerName="main"
Sep 18 19:07:43.719140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.718999    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" containerName="main"
Sep 18 19:07:43.719140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.719098    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="709e318c-09aa-465d-a108-9cb6833371ea" containerName="payload-pre-processing"
Sep 18 19:07:43.719140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.719110    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa30389-314c-4369-a0ac-84cbca9a0e5f" containerName="payload-processing"
Sep 18 19:07:43.719140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.719126    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc1fb6dd-6a5c-47d6-a7c0-dc9fd377e016" containerName="main"
Sep 18 19:07:43.724064 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.724038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.732768 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.732735    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-feb80d-serving-cert\""
Sep 18 19:07:43.742089 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.742060    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"]
Sep 18 19:07:43.757313 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.757275    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:43.852378 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.852333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:43.874489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.874453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.874489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.874502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.874750 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.874530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.874750 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.874654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlk9\" (UniqueName: \"kubernetes.io/projected/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-kube-api-access-xhlk9\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.976100 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.975989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlk9\" (UniqueName: \"kubernetes.io/projected/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-kube-api-access-xhlk9\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.976100 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.976089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.976354 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.976125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.976354 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.976152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.976354 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:07:43.976244    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-serving-cert: secret "maas-api-e2e-shared-a-w3-feb80d-serving-cert" not found
Sep 18 19:07:43.976354 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:07:43.976337    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls podName:e0f9d0cb-02fe-437f-93c2-4b2299ec2122 nodeName:}" failed. No retries permitted until 2026-09-18 19:07:44.476314778 +0000 UTC m=+1388.268991687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls") pod "maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" (UID: "e0f9d0cb-02fe-437f-93c2-4b2299ec2122") : secret "maas-api-e2e-shared-a-w3-feb80d-serving-cert" not found
Sep 18 19:07:43.977008 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.976982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.978792 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.978772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:43.984850 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:43.984826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlk9\" (UniqueName: \"kubernetes.io/projected/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-kube-api-access-xhlk9\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:44.288007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.287916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:44.417314 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.417279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:44.481600 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.481562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:44.484301 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.484280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:44.572191 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.572106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:44.645929 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.645864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:44.925528 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.925496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"]
Sep 18 19:07:44.928190 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:07:44.928163    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f9d0cb_02fe_437f_93c2_4b2299ec2122.slice/crio-b09cb8299a64843078b94c5a48e9434a9db0ceecf2424a9000c4516e512dac50 WatchSource:0}: Error finding container b09cb8299a64843078b94c5a48e9434a9db0ceecf2424a9000c4516e512dac50: Status 404 returned error can't find the container with id b09cb8299a64843078b94c5a48e9434a9db0ceecf2424a9000c4516e512dac50
Sep 18 19:07:44.929507 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:44.929488    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:07:45.869710 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:45.869601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" event={"ID":"e0f9d0cb-02fe-437f-93c2-4b2299ec2122","Type":"ContainerStarted","Data":"202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f"}
Sep 18 19:07:45.869710 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:45.869659    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" event={"ID":"e0f9d0cb-02fe-437f-93c2-4b2299ec2122","Type":"ContainerStarted","Data":"b09cb8299a64843078b94c5a48e9434a9db0ceecf2424a9000c4516e512dac50"}
Sep 18 19:07:45.869710 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:45.869680    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:45.890251 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:45.890199    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" podStartSLOduration=2.381620335 podStartE2EDuration="2.890179697s" totalImagesPullingTime="508.559362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:07:45.888844534 +0000 UTC m=+1389.681521489" watchObservedRunningTime="2026-09-18 19:07:45.890179697 +0000 UTC m=+1389.682856628"
Sep 18 19:07:48.376102 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:48.376064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:51.881489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:51.881456    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:07:52.075745 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.075710    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"]
Sep 18 19:07:52.079271 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.079254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.082424 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.082390    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4ac9ac7d\"/\"kube-root-ca.crt\""
Sep 18 19:07:52.082424 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.082404    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4ac9ac7d\"/\"openshift-service-ca.crt\""
Sep 18 19:07:52.082586 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.082439    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4ac9ac7d\"/\"e2e-lifecycle-model-4ac9ac7d-kserve-self-signed-certs\""
Sep 18 19:07:52.082586 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.082470    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4ac9ac7d\"/\"default-dockercfg-v8rqr\""
Sep 18 19:07:52.090439 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.090416    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"]
Sep 18 19:07:52.246304 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.246268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-dshm\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.246474 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.246308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tmp-dir\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.246474 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.246373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvbd\" (UniqueName: \"kubernetes.io/projected/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-kube-api-access-ncvbd\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.246474 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.246401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-home\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.246474 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.246455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tls-certs\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.246613 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.246489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-model-cache\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347198 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-dshm\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347198 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tmp-dir\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347609 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvbd\" (UniqueName: \"kubernetes.io/projected/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-kube-api-access-ncvbd\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347609 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-home\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347609 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tls-certs\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347609 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-model-cache\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347872 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-home\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347872 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tmp-dir\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.347872 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.347782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-model-cache\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.349748 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.349718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-dshm\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.350071 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.350053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tls-certs\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.356812 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.356775    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvbd\" (UniqueName: \"kubernetes.io/projected/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-kube-api-access-ncvbd\") pod \"e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") " pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.388828 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.388786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:07:52.395859 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.395819    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:07:52.675787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.675754    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"]
Sep 18 19:07:52.679844 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:07:52.679812    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod616a7fd9_5ca8_4e12_82dd_83a577ab09f0.slice/crio-fe0843d161ccc4908524aa6bca43d1a99c6df5b3f405099e999809ce02dd8bdc WatchSource:0}: Error finding container fe0843d161ccc4908524aa6bca43d1a99c6df5b3f405099e999809ce02dd8bdc: Status 404 returned error can't find the container with id fe0843d161ccc4908524aa6bca43d1a99c6df5b3f405099e999809ce02dd8bdc
Sep 18 19:07:52.697659 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.697569    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:07:52.912134 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.912089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz" event={"ID":"616a7fd9-5ca8-4e12-82dd-83a577ab09f0","Type":"ContainerStarted","Data":"53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261"}
Sep 18 19:07:52.912536 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.912155    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz" event={"ID":"616a7fd9-5ca8-4e12-82dd-83a577ab09f0","Type":"ContainerStarted","Data":"fe0843d161ccc4908524aa6bca43d1a99c6df5b3f405099e999809ce02dd8bdc"}
Sep 18 19:07:52.935402 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:52.935303    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz" podStartSLOduration=0.935289658 podStartE2EDuration="935.289658ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:07:52.93384496 +0000 UTC m=+1396.726521908" watchObservedRunningTime="2026-09-18 19:07:52.935289658 +0000 UTC m=+1396.727966590"
Sep 18 19:07:57.255625 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:57.255590    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"]
Sep 18 19:07:57.306027 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:07:57.305987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:00.104489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:00.104450    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"]
Sep 18 19:08:00.138580 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:00.138555    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:02.389754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:02.389719    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:08:02.390197 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:02.389768    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:08:02.402274 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:02.402251    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:08:02.973859 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:02.973826    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:08:06.970366 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:06.970325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:07.832490 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:07.832453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:08.539932 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:08.539901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:08.739542 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:08.739503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:08.869074 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:08.868989    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:09.240689 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:09.240653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:09.801135 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:09.801096    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:10.000263 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:10.000234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:11.032908 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:11.032848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:12.275283 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.275238    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86" podUID="650e2b95-7549-4a90-a1df-06257f3a8a2c" containerName="main" containerID="cri-o://907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7" gracePeriod=45
Sep 18 19:08:12.342968 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.342936    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:12.473606 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.473580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:12.531135 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.531074    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:08:12.622191 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622164    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-model-cache\") pod \"650e2b95-7549-4a90-a1df-06257f3a8a2c\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") "
Sep 18 19:08:12.622191 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622197    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650e2b95-7549-4a90-a1df-06257f3a8a2c-tls-certs\") pod \"650e2b95-7549-4a90-a1df-06257f3a8a2c\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") "
Sep 18 19:08:12.622408 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622249    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-tmp-dir\") pod \"650e2b95-7549-4a90-a1df-06257f3a8a2c\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") "
Sep 18 19:08:12.622408 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-home\") pod \"650e2b95-7549-4a90-a1df-06257f3a8a2c\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") "
Sep 18 19:08:12.622408 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj28z\" (UniqueName: \"kubernetes.io/projected/650e2b95-7549-4a90-a1df-06257f3a8a2c-kube-api-access-qj28z\") pod \"650e2b95-7549-4a90-a1df-06257f3a8a2c\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") "
Sep 18 19:08:12.622408 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-dshm\") pod \"650e2b95-7549-4a90-a1df-06257f3a8a2c\" (UID: \"650e2b95-7549-4a90-a1df-06257f3a8a2c\") "
Sep 18 19:08:12.622622 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "650e2b95-7549-4a90-a1df-06257f3a8a2c" (UID: "650e2b95-7549-4a90-a1df-06257f3a8a2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:12.622622 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622541    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "650e2b95-7549-4a90-a1df-06257f3a8a2c" (UID: "650e2b95-7549-4a90-a1df-06257f3a8a2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:12.622622 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622616    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-home" (OuterVolumeSpecName: "home") pod "650e2b95-7549-4a90-a1df-06257f3a8a2c" (UID: "650e2b95-7549-4a90-a1df-06257f3a8a2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:12.622778 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622757    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:12.622837 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622777    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:12.622837 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.622792    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:12.624648 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.624627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-dshm" (OuterVolumeSpecName: "dshm") pod "650e2b95-7549-4a90-a1df-06257f3a8a2c" (UID: "650e2b95-7549-4a90-a1df-06257f3a8a2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:12.624735 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.624695    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650e2b95-7549-4a90-a1df-06257f3a8a2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "650e2b95-7549-4a90-a1df-06257f3a8a2c" (UID: "650e2b95-7549-4a90-a1df-06257f3a8a2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:08:12.624776 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.624736    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650e2b95-7549-4a90-a1df-06257f3a8a2c-kube-api-access-qj28z" (OuterVolumeSpecName: "kube-api-access-qj28z") pod "650e2b95-7549-4a90-a1df-06257f3a8a2c" (UID: "650e2b95-7549-4a90-a1df-06257f3a8a2c"). InnerVolumeSpecName "kube-api-access-qj28z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:08:12.723613 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.723587    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/650e2b95-7549-4a90-a1df-06257f3a8a2c-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:12.723613 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.723608    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj28z\" (UniqueName: \"kubernetes.io/projected/650e2b95-7549-4a90-a1df-06257f3a8a2c-kube-api-access-qj28z\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:12.723751 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:12.723618    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/650e2b95-7549-4a90-a1df-06257f3a8a2c-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:13.016680 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.016651    2582 generic.go:356] "Generic (PLEG): container finished" podID="650e2b95-7549-4a90-a1df-06257f3a8a2c" containerID="907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7" exitCode=0
Sep 18 19:08:13.016920 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.016713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86" event={"ID":"650e2b95-7549-4a90-a1df-06257f3a8a2c","Type":"ContainerDied","Data":"907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7"}
Sep 18 19:08:13.016920 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.016746    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86" event={"ID":"650e2b95-7549-4a90-a1df-06257f3a8a2c","Type":"ContainerDied","Data":"1eceff5cd9e2ac4f7c17fb8eb502df4d6ae795ad48cd4e82ae8ce091fa18de3d"}
Sep 18 19:08:13.016920 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.016769    2582 scope.go:117] "RemoveContainer" containerID="907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7"
Sep 18 19:08:13.016920 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.016771    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"
Sep 18 19:08:13.029504 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.029478    2582 scope.go:117] "RemoveContainer" containerID="907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7"
Sep 18 19:08:13.030070 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:08:13.030047    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7\": container with ID starting with 907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7 not found: ID does not exist" containerID="907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7"
Sep 18 19:08:13.030180 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.030075    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7"} err="failed to get container status \"907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7\": rpc error: code = NotFound desc = could not find container \"907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7\": container with ID starting with 907edab687c282ff08dccae4ac7126f317ada2a5babef0c184109a32f5f89de7 not found: ID does not exist"
Sep 18 19:08:13.062672 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.062640    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"]
Sep 18 19:08:13.069413 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.069392    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-b-af656e27-kserve-8476c9d5bc-glb86"]
Sep 18 19:08:13.739574 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:13.739537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:14.335811 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.335769    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="650e2b95-7549-4a90-a1df-06257f3a8a2c" path="/var/lib/kubelet/pods/650e2b95-7549-4a90-a1df-06257f3a8a2c/volumes"
Sep 18 19:08:14.759405 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.759367    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"]
Sep 18 19:08:14.760132 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.760026    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="650e2b95-7549-4a90-a1df-06257f3a8a2c" containerName="main"
Sep 18 19:08:14.760132 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.760047    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="650e2b95-7549-4a90-a1df-06257f3a8a2c" containerName="main"
Sep 18 19:08:14.760255 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.760155    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="650e2b95-7549-4a90-a1df-06257f3a8a2c" containerName="main"
Sep 18 19:08:14.764728 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.764705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.768107 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.768082    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-19944d-serving-cert\""
Sep 18 19:08:14.785194 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.785172    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"]
Sep 18 19:08:14.827620 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.827592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:14.842113 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.842088    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84wx\" (UniqueName: \"kubernetes.io/projected/36821d1a-405e-4a1c-b66c-b53460494109-kube-api-access-v84wx\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.842246 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.842156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36821d1a-405e-4a1c-b66c-b53460494109-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.842246 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.842202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.842246 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.842227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.943091 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.943062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84wx\" (UniqueName: \"kubernetes.io/projected/36821d1a-405e-4a1c-b66c-b53460494109-kube-api-access-v84wx\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.943229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.943117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36821d1a-405e-4a1c-b66c-b53460494109-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.943229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.943158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.943229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.943176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.943935 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.943914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36821d1a-405e-4a1c-b66c-b53460494109-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.945871 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.945844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.945982 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.945968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.954750 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.954727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84wx\" (UniqueName: \"kubernetes.io/projected/36821d1a-405e-4a1c-b66c-b53460494109-kube-api-access-v84wx\") pod \"maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:14.968137 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:14.968117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:15.066401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.066324    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:15.076439 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.076419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:15.124287 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.124069    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn" podUID="87fbdaa7-eab6-4962-9c16-ff06b9812fd0" containerName="main" containerID="cri-o://255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66" gracePeriod=45
Sep 18 19:08:15.356046 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.356021    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:08:15.366805 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.366785    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"]
Sep 18 19:08:15.368907 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:08:15.368866    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36821d1a_405e_4a1c_b66c_b53460494109.slice/crio-b5b30d3596123837eb8f3cbec39efa99d974d2afafd66bd0908002a6059d6028 WatchSource:0}: Error finding container b5b30d3596123837eb8f3cbec39efa99d974d2afafd66bd0908002a6059d6028: Status 404 returned error can't find the container with id b5b30d3596123837eb8f3cbec39efa99d974d2afafd66bd0908002a6059d6028
Sep 18 19:08:15.412140 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.412110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:15.447484 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447460    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tmp-dir\") pod \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") "
Sep 18 19:08:15.447579 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-dshm\") pod \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") "
Sep 18 19:08:15.447579 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447567    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-model-cache\") pod \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") "
Sep 18 19:08:15.447697 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tls-certs\") pod \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") "
Sep 18 19:08:15.447697 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdm7k\" (UniqueName: \"kubernetes.io/projected/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-kube-api-access-kdm7k\") pod \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") "
Sep 18 19:08:15.447697 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447646    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-home\") pod \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\" (UID: \"87fbdaa7-eab6-4962-9c16-ff06b9812fd0\") "
Sep 18 19:08:15.447857 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87fbdaa7-eab6-4962-9c16-ff06b9812fd0" (UID: "87fbdaa7-eab6-4962-9c16-ff06b9812fd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:15.447857 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "87fbdaa7-eab6-4962-9c16-ff06b9812fd0" (UID: "87fbdaa7-eab6-4962-9c16-ff06b9812fd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:15.448020 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.447994    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-home" (OuterVolumeSpecName: "home") pod "87fbdaa7-eab6-4962-9c16-ff06b9812fd0" (UID: "87fbdaa7-eab6-4962-9c16-ff06b9812fd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:15.448365 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.448213    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:15.448365 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.448239    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:15.448365 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.448253    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:15.450097 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.450073    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-kube-api-access-kdm7k" (OuterVolumeSpecName: "kube-api-access-kdm7k") pod "87fbdaa7-eab6-4962-9c16-ff06b9812fd0" (UID: "87fbdaa7-eab6-4962-9c16-ff06b9812fd0"). InnerVolumeSpecName "kube-api-access-kdm7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:08:15.450339 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.450321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-dshm" (OuterVolumeSpecName: "dshm") pod "87fbdaa7-eab6-4962-9c16-ff06b9812fd0" (UID: "87fbdaa7-eab6-4962-9c16-ff06b9812fd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:08:15.450339 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.450331    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87fbdaa7-eab6-4962-9c16-ff06b9812fd0" (UID: "87fbdaa7-eab6-4962-9c16-ff06b9812fd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:08:15.549270 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.549243    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:15.549270 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.549266    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:15.549270 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.549277    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdm7k\" (UniqueName: \"kubernetes.io/projected/87fbdaa7-eab6-4962-9c16-ff06b9812fd0-kube-api-access-kdm7k\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:15.917259 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:15.917220    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:16.005492 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.005460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:16.034800 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.034769    2582 generic.go:356] "Generic (PLEG): container finished" podID="87fbdaa7-eab6-4962-9c16-ff06b9812fd0" containerID="255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66" exitCode=0
Sep 18 19:08:16.034990 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.034809    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"
Sep 18 19:08:16.034990 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.034857    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn" event={"ID":"87fbdaa7-eab6-4962-9c16-ff06b9812fd0","Type":"ContainerDied","Data":"255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66"}
Sep 18 19:08:16.034990 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.034928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn" event={"ID":"87fbdaa7-eab6-4962-9c16-ff06b9812fd0","Type":"ContainerDied","Data":"8816a85fb65ab1fd31f97b7927aef37a99fbb7b52982f809899fffbafdeddf40"}
Sep 18 19:08:16.034990 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.034948    2582 scope.go:117] "RemoveContainer" containerID="255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66"
Sep 18 19:08:16.036815 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.036780    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz" event={"ID":"36821d1a-405e-4a1c-b66c-b53460494109","Type":"ContainerStarted","Data":"6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1"}
Sep 18 19:08:16.036943 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.036831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz" event={"ID":"36821d1a-405e-4a1c-b66c-b53460494109","Type":"ContainerStarted","Data":"b5b30d3596123837eb8f3cbec39efa99d974d2afafd66bd0908002a6059d6028"}
Sep 18 19:08:16.036943 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.036861    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:16.044563 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.044547    2582 scope.go:117] "RemoveContainer" containerID="255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66"
Sep 18 19:08:16.044831 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:08:16.044810    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66\": container with ID starting with 255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66 not found: ID does not exist" containerID="255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66"
Sep 18 19:08:16.044921 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.044844    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66"} err="failed to get container status \"255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66\": rpc error: code = NotFound desc = could not find container \"255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66\": container with ID starting with 255fe6778df3e0652a319c073de86d42eb09a782501b799e9b9d1403092c7a66 not found: ID does not exist"
Sep 18 19:08:16.081489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.081447    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz" podStartSLOduration=1.76239517 podStartE2EDuration="2.081433473s" totalImagesPullingTime="319.038303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:08:16.080180208 +0000 UTC m=+1419.872857157" watchObservedRunningTime="2026-09-18 19:08:16.081433473 +0000 UTC m=+1419.874110403"
Sep 18 19:08:16.124303 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.124272    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"]
Sep 18 19:08:16.137031 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.137002    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-942883/e2e-conflict-a-af656e27-kserve-5c48864585-fp4gn"]
Sep 18 19:08:16.335223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.335143    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fbdaa7-eab6-4962-9c16-ff06b9812fd0" path="/var/lib/kubelet/pods/87fbdaa7-eab6-4962-9c16-ff06b9812fd0/volumes"
Sep 18 19:08:16.571803 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:16.571771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:17.971659 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:17.971624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:19.170416 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:19.170337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:20.133080 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:20.133043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:21.053313 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:21.053281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:21.997102 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:21.997074    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:22.049307 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:22.049285    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:08:23.940713 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:23.940669    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:24.923330 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:24.923270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:25.801600 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:25.801568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:26.745642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:26.745614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:26.914117 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:26.914078    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:08:27.123754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:27.123667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:28.010155 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:28.010126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:29.013610 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:29.013576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:30.368244 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:30.368211    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:31.478086 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:31.478052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:33.603205 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:33.603169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:34.986645 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:34.986613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:35.980815 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:35.980767    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:36.867405 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:36.867373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:37.828625 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:37.828588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:38.812863 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:38.812789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:40.178155 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:40.178113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:41.250105 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:41.250059    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:42.139644 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:42.139610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:48.619902 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.619855    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"]
Sep 18 19:08:48.620277 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.620267    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fbdaa7-eab6-4962-9c16-ff06b9812fd0" containerName="main"
Sep 18 19:08:48.620320 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.620279    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fbdaa7-eab6-4962-9c16-ff06b9812fd0" containerName="main"
Sep 18 19:08:48.620356 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.620347    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fbdaa7-eab6-4962-9c16-ff06b9812fd0" containerName="main"
Sep 18 19:08:48.624415 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.624398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.627484 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.627462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"kube-root-ca.crt\""
Sep 18 19:08:48.627593 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.627483    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"openshift-service-ca.crt\""
Sep 18 19:08:48.628242 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.628226    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"auth-test-model-feb80d-kserve-self-signed-certs\""
Sep 18 19:08:48.629488 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.629466    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"default-dockercfg-5f268\""
Sep 18 19:08:48.645828 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.645808    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"]
Sep 18 19:08:48.655855 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.655832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-home\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.655987 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.655930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-dshm\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.655987 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.655952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-model-cache\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.655987 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.655971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tmp-dir\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.656163 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.656068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnqg\" (UniqueName: \"kubernetes.io/projected/6237c2fc-8ac5-4148-b39c-60ed7c072d37-kube-api-access-7vnqg\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.656163 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.656116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tls-certs\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.698038 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.698015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:48.757072 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-dshm\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757180 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-model-cache\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757180 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tmp-dir\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757180 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnqg\" (UniqueName: \"kubernetes.io/projected/6237c2fc-8ac5-4148-b39c-60ed7c072d37-kube-api-access-7vnqg\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757356 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tls-certs\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757356 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-home\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757483 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-model-cache\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757542 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757528    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tmp-dir\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.757625 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.757609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-home\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.759391 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.759369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-dshm\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.759737 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.759719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tls-certs\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.768518 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.768499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnqg\" (UniqueName: \"kubernetes.io/projected/6237c2fc-8ac5-4148-b39c-60ed7c072d37-kube-api-access-7vnqg\") pod \"auth-test-model-feb80d-kserve-5cc8db768c-7fbdr\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:48.934186 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:48.934160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:49.237214 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.237130    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"]
Sep 18 19:08:49.241097 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:08:49.241063    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6237c2fc_8ac5_4148_b39c_60ed7c072d37.slice/crio-58a514c64d3bfcd57a47b6d4a75e6e4207c781796442fe9b2bbce65d05bfd14b WatchSource:0}: Error finding container 58a514c64d3bfcd57a47b6d4a75e6e4207c781796442fe9b2bbce65d05bfd14b: Status 404 returned error can't find the container with id 58a514c64d3bfcd57a47b6d4a75e6e4207c781796442fe9b2bbce65d05bfd14b
Sep 18 19:08:49.352983 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.352959    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:49.415278 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.415248    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"]
Sep 18 19:08:49.415508 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.415486    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t" podUID="36a823c2-2227-4fcb-82a2-881ee1672aec" containerName="payload-pre-processing" containerID="cri-o://945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994" gracePeriod=30
Sep 18 19:08:49.560687 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.560653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:49.651180 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.651154    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:08:49.664684 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.664648    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgkmc\" (UniqueName: \"kubernetes.io/projected/36a823c2-2227-4fcb-82a2-881ee1672aec-kube-api-access-mgkmc\") pod \"36a823c2-2227-4fcb-82a2-881ee1672aec\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") "
Sep 18 19:08:49.664825 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.664695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36a823c2-2227-4fcb-82a2-881ee1672aec-plugins-config-volume\") pod \"36a823c2-2227-4fcb-82a2-881ee1672aec\" (UID: \"36a823c2-2227-4fcb-82a2-881ee1672aec\") "
Sep 18 19:08:49.665143 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.665122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a823c2-2227-4fcb-82a2-881ee1672aec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36a823c2-2227-4fcb-82a2-881ee1672aec" (UID: "36a823c2-2227-4fcb-82a2-881ee1672aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:08:49.666868 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.666817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a823c2-2227-4fcb-82a2-881ee1672aec-kube-api-access-mgkmc" (OuterVolumeSpecName: "kube-api-access-mgkmc") pod "36a823c2-2227-4fcb-82a2-881ee1672aec" (UID: "36a823c2-2227-4fcb-82a2-881ee1672aec"). InnerVolumeSpecName "kube-api-access-mgkmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:08:49.765763 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.765733    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgkmc\" (UniqueName: \"kubernetes.io/projected/36a823c2-2227-4fcb-82a2-881ee1672aec-kube-api-access-mgkmc\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:49.765763 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.765763    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36a823c2-2227-4fcb-82a2-881ee1672aec-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:08:49.849532 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:49.849463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:50.024985 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.024963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:50.241435 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.241399    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr" event={"ID":"6237c2fc-8ac5-4148-b39c-60ed7c072d37","Type":"ContainerStarted","Data":"8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5"}
Sep 18 19:08:50.241583 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.241447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr" event={"ID":"6237c2fc-8ac5-4148-b39c-60ed7c072d37","Type":"ContainerStarted","Data":"58a514c64d3bfcd57a47b6d4a75e6e4207c781796442fe9b2bbce65d05bfd14b"}
Sep 18 19:08:50.242787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.242762    2582 generic.go:356] "Generic (PLEG): container finished" podID="36a823c2-2227-4fcb-82a2-881ee1672aec" containerID="945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994" exitCode=0
Sep 18 19:08:50.242926 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.242799    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"
Sep 18 19:08:50.242926 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.242826    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t" event={"ID":"36a823c2-2227-4fcb-82a2-881ee1672aec","Type":"ContainerDied","Data":"945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994"}
Sep 18 19:08:50.242926 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.242866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t" event={"ID":"36a823c2-2227-4fcb-82a2-881ee1672aec","Type":"ContainerDied","Data":"1bc0180943d0207bf063395758e0789c98c3d41df48d73dc302613c101348a02"}
Sep 18 19:08:50.242926 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.242907    2582 scope.go:117] "RemoveContainer" containerID="945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994"
Sep 18 19:08:50.253334 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.253314    2582 scope.go:117] "RemoveContainer" containerID="945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994"
Sep 18 19:08:50.253608 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:08:50.253588    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994\": container with ID starting with 945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994 not found: ID does not exist" containerID="945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994"
Sep 18 19:08:50.253656 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.253617    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994"} err="failed to get container status \"945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994\": rpc error: code = NotFound desc = could not find container \"945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994\": container with ID starting with 945871b31d1403413541e75c8b82eb5589b8d7fbe192a15c653999321182e994 not found: ID does not exist"
Sep 18 19:08:50.275601 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.275556    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr" podStartSLOduration=2.275542748 podStartE2EDuration="2.275542748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:08:50.273187774 +0000 UTC m=+1454.065864735" watchObservedRunningTime="2026-09-18 19:08:50.275542748 +0000 UTC m=+1454.068219732"
Sep 18 19:08:50.308296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.308265    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"]
Sep 18 19:08:50.325803 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.325781    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4ac9ac7d-698cdc48bb-2lc5t"]
Sep 18 19:08:50.335449 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.335424    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a823c2-2227-4fcb-82a2-881ee1672aec" path="/var/lib/kubelet/pods/36a823c2-2227-4fcb-82a2-881ee1672aec/volumes"
Sep 18 19:08:50.577508 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:50.577446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:51.790533 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:51.790500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:53.077225 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:53.077193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:53.609872 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:53.609837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:54.455361 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:54.455327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:55.596702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:55.596672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:56.995749 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:56.995718    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:58.221300 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:58.221268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:58.636918 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:58.636806    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"]
Sep 18 19:08:58.732681 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:58.732652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:08:58.934432 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:58.934403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:58.934615 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:58.934447    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:58.947077 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:58.947054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:08:59.306024 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:08:59.305945    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:09:13.291965 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.291922    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:09:13.659067 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.658994    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz" podUID="616a7fd9-5ca8-4e12-82dd-83a577ab09f0" containerName="main" containerID="cri-o://53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261" gracePeriod=45
Sep 18 19:09:13.924081 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.923995    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:09:13.983018 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.982985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvbd\" (UniqueName: \"kubernetes.io/projected/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-kube-api-access-ncvbd\") pod \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") "
Sep 18 19:09:13.983223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983061    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tmp-dir\") pod \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") "
Sep 18 19:09:13.983223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983092    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-home\") pod \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") "
Sep 18 19:09:13.983223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-model-cache\") pod \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") "
Sep 18 19:09:13.983223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983178    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tls-certs\") pod \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") "
Sep 18 19:09:13.983462 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983241    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-dshm\") pod \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\" (UID: \"616a7fd9-5ca8-4e12-82dd-83a577ab09f0\") "
Sep 18 19:09:13.983462 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "616a7fd9-5ca8-4e12-82dd-83a577ab09f0" (UID: "616a7fd9-5ca8-4e12-82dd-83a577ab09f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.983462 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-home" (OuterVolumeSpecName: "home") pod "616a7fd9-5ca8-4e12-82dd-83a577ab09f0" (UID: "616a7fd9-5ca8-4e12-82dd-83a577ab09f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.983592 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "616a7fd9-5ca8-4e12-82dd-83a577ab09f0" (UID: "616a7fd9-5ca8-4e12-82dd-83a577ab09f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.983724 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983704    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.983769 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983731    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.983769 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.983746    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:09:13.985574 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.985551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-dshm" (OuterVolumeSpecName: "dshm") pod "616a7fd9-5ca8-4e12-82dd-83a577ab09f0" (UID: "616a7fd9-5ca8-4e12-82dd-83a577ab09f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:09:13.985686 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.985597    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-kube-api-access-ncvbd" (OuterVolumeSpecName: "kube-api-access-ncvbd") pod "616a7fd9-5ca8-4e12-82dd-83a577ab09f0" (UID: "616a7fd9-5ca8-4e12-82dd-83a577ab09f0"). InnerVolumeSpecName "kube-api-access-ncvbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:09:13.985914 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:13.985869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "616a7fd9-5ca8-4e12-82dd-83a577ab09f0" (UID: "616a7fd9-5ca8-4e12-82dd-83a577ab09f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:09:14.085410 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.085362    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:09:14.085410 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.085411    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvbd\" (UniqueName: \"kubernetes.io/projected/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-kube-api-access-ncvbd\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:09:14.085635 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.085427    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/616a7fd9-5ca8-4e12-82dd-83a577ab09f0-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:09:14.376326 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.376279    2582 generic.go:356] "Generic (PLEG): container finished" podID="616a7fd9-5ca8-4e12-82dd-83a577ab09f0" containerID="53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261" exitCode=0
Sep 18 19:09:14.376326 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.376323    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"
Sep 18 19:09:14.376935 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.376334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz" event={"ID":"616a7fd9-5ca8-4e12-82dd-83a577ab09f0","Type":"ContainerDied","Data":"53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261"}
Sep 18 19:09:14.376935 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.376380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz" event={"ID":"616a7fd9-5ca8-4e12-82dd-83a577ab09f0","Type":"ContainerDied","Data":"fe0843d161ccc4908524aa6bca43d1a99c6df5b3f405099e999809ce02dd8bdc"}
Sep 18 19:09:14.376935 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.376398    2582 scope.go:117] "RemoveContainer" containerID="53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261"
Sep 18 19:09:14.386775 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.386746    2582 scope.go:117] "RemoveContainer" containerID="53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261"
Sep 18 19:09:14.387238 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:09:14.387202    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261\": container with ID starting with 53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261 not found: ID does not exist" containerID="53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261"
Sep 18 19:09:14.387349 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.387246    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261"} err="failed to get container status \"53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261\": rpc error: code = NotFound desc = could not find container \"53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261\": container with ID starting with 53f519426222f251098cf075b83c7081af0c357deecada7abc42f4e928b37261 not found: ID does not exist"
Sep 18 19:09:14.404934 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.404867    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"]
Sep 18 19:09:14.412315 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:14.412279    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4ac9ac7d/e2e-lifecycle-model-4ac9ac7d-kserve-68bfb89d4c-9g6pz"]
Sep 18 19:09:16.335141 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:16.335108    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616a7fd9-5ca8-4e12-82dd-83a577ab09f0" path="/var/lib/kubelet/pods/616a7fd9-5ca8-4e12-82dd-83a577ab09f0/volumes"
Sep 18 19:09:22.007171 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.007134    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"]
Sep 18 19:09:22.007564 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.007523    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a823c2-2227-4fcb-82a2-881ee1672aec" containerName="payload-pre-processing"
Sep 18 19:09:22.007564 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.007534    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a823c2-2227-4fcb-82a2-881ee1672aec" containerName="payload-pre-processing"
Sep 18 19:09:22.007564 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.007552    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616a7fd9-5ca8-4e12-82dd-83a577ab09f0" containerName="main"
Sep 18 19:09:22.007564 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.007557    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="616a7fd9-5ca8-4e12-82dd-83a577ab09f0" containerName="main"
Sep 18 19:09:22.007705 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.007626    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="616a7fd9-5ca8-4e12-82dd-83a577ab09f0" containerName="main"
Sep 18 19:09:22.007705 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.007636    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a823c2-2227-4fcb-82a2-881ee1672aec" containerName="payload-pre-processing"
Sep 18 19:09:22.012011 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.011994    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.014936 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.014916    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-942883\"/\"distinct-2-w4-942883-kserve-self-signed-certs\""
Sep 18 19:09:22.028583 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.028560    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"]
Sep 18 19:09:22.098580 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.098553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:22.158995 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.158968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqpsb\" (UniqueName: \"kubernetes.io/projected/4ea0e6e9-ec57-405f-8b53-b698068e8edf-kube-api-access-xqpsb\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.159098 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.158998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-home\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.159098 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.159021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-dshm\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.159205 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.159182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-model-cache\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.159245 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.159221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tls-certs\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.159290 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.159252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tmp-dir\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.260787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.260704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-model-cache\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.260787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.260749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tls-certs\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.260787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.260766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tmp-dir\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.260787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.260788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqpsb\" (UniqueName: \"kubernetes.io/projected/4ea0e6e9-ec57-405f-8b53-b698068e8edf-kube-api-access-xqpsb\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.261092 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.260808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-home\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.261092 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.260837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-dshm\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.261202 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.261136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-model-cache\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.261254 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.261216    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tmp-dir\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.261314 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.261259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-home\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.263362 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.263340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-dshm\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.263594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.263574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tls-certs\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.271869 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.271844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqpsb\" (UniqueName: \"kubernetes.io/projected/4ea0e6e9-ec57-405f-8b53-b698068e8edf-kube-api-access-xqpsb\") pod \"distinct-2-w4-942883-kserve-74b786569d-5qr68\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") " pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.321372 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.321324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:22.596165 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:22.596135    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"]
Sep 18 19:09:22.598251 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:09:22.598223    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea0e6e9_ec57_405f_8b53_b698068e8edf.slice/crio-c52d4f5f3159cc883b6bcb2d7457ccc5a0633b5d3ce487f7fa23412e305a157a WatchSource:0}: Error finding container c52d4f5f3159cc883b6bcb2d7457ccc5a0633b5d3ce487f7fa23412e305a157a: Status 404 returned error can't find the container with id c52d4f5f3159cc883b6bcb2d7457ccc5a0633b5d3ce487f7fa23412e305a157a
Sep 18 19:09:23.433942 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:23.433907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68" event={"ID":"4ea0e6e9-ec57-405f-8b53-b698068e8edf","Type":"ContainerStarted","Data":"180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be"}
Sep 18 19:09:23.433942 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:23.433949    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68" event={"ID":"4ea0e6e9-ec57-405f-8b53-b698068e8edf","Type":"ContainerStarted","Data":"c52d4f5f3159cc883b6bcb2d7457ccc5a0633b5d3ce487f7fa23412e305a157a"}
Sep 18 19:09:23.465117 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:23.465067    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68" podStartSLOduration=2.465049487 podStartE2EDuration="2.465049487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:09:23.461656189 +0000 UTC m=+1487.254333121" watchObservedRunningTime="2026-09-18 19:09:23.465049487 +0000 UTC m=+1487.257726417"
Sep 18 19:09:32.322188 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:32.322155    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:32.322656 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:32.322201    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:32.336841 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:32.336815    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:32.493469 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:32.493442    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:09:33.591937 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:33.591906    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:35.757926 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:35.757900    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:37.420496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:37.420463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:37.670759 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:37.670678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:43.429548 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:43.429512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:45.354172 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:45.354143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:09:54.720532 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.720453    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"]
Sep 18 19:09:54.724218 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.724202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.727498 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.727476    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-19944d\"/\"default-dockercfg-mqls5\""
Sep 18 19:09:54.727642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.727566    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-19944d\"/\"auth-test-model-19944d-kserve-self-signed-certs\""
Sep 18 19:09:54.727642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.727600    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-19944d\"/\"openshift-service-ca.crt\""
Sep 18 19:09:54.727642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.727571    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-19944d\"/\"kube-root-ca.crt\""
Sep 18 19:09:54.741312 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.741290    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"]
Sep 18 19:09:54.757564 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.757530    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:54.873037 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.873003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lqxk\" (UniqueName: \"kubernetes.io/projected/cb60599d-3543-458f-b8be-a9b7e6b988be-kube-api-access-2lqxk\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.873219 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.873044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb60599d-3543-458f-b8be-a9b7e6b988be-tls-certs\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.873219 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.873110    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-dshm\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.873219 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.873162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-home\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.873219 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.873188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-model-cache\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.873219 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.873208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-tmp-dir\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974062 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.973986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-model-cache\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974062 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-tmp-dir\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974062 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lqxk\" (UniqueName: \"kubernetes.io/projected/cb60599d-3543-458f-b8be-a9b7e6b988be-kube-api-access-2lqxk\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974301 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb60599d-3543-458f-b8be-a9b7e6b988be-tls-certs\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974301 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-dshm\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974301 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-home\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974502 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-model-cache\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974565 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-tmp-dir\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.974565 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.974520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-home\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.976603 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.976575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-dshm\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.977036 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.977016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb60599d-3543-458f-b8be-a9b7e6b988be-tls-certs\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:54.984344 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:54.984323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lqxk\" (UniqueName: \"kubernetes.io/projected/cb60599d-3543-458f-b8be-a9b7e6b988be-kube-api-access-2lqxk\") pod \"auth-test-model-19944d-kserve-69f67c65ff-mrxrp\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") " pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:55.033335 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:55.033313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:09:55.308363 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:55.308330    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"]
Sep 18 19:09:55.309990 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:09:55.309962    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb60599d_3543_458f_b8be_a9b7e6b988be.slice/crio-aa70539a78bd4df840d8effe5afd41eeeb869b26ae2f24b53f72be16b3618507 WatchSource:0}: Error finding container aa70539a78bd4df840d8effe5afd41eeeb869b26ae2f24b53f72be16b3618507: Status 404 returned error can't find the container with id aa70539a78bd4df840d8effe5afd41eeeb869b26ae2f24b53f72be16b3618507
Sep 18 19:09:55.611316 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:55.611216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp" event={"ID":"cb60599d-3543-458f-b8be-a9b7e6b988be","Type":"ContainerStarted","Data":"003f0105a71d772d6d25c2f6ece1c4dc991f3cd144a486fa366f43ed45ef62ba"}
Sep 18 19:09:55.611316 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:55.611271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp" event={"ID":"cb60599d-3543-458f-b8be-a9b7e6b988be","Type":"ContainerStarted","Data":"aa70539a78bd4df840d8effe5afd41eeeb869b26ae2f24b53f72be16b3618507"}
Sep 18 19:09:55.629875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:55.629823    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp" podStartSLOduration=1.629807729 podStartE2EDuration="1.629807729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:09:55.629128061 +0000 UTC m=+1519.421805018" watchObservedRunningTime="2026-09-18 19:09:55.629807729 +0000 UTC m=+1519.422484713"
Sep 18 19:09:58.833390 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:58.833356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:58.925988 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:58.925957    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:59.166088 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:59.166056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:09:59.342102 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:09:59.342072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:00.530029 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:00.529993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:00.688025 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:00.687994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:01.603090 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:01.603061    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:02.029009 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:02.028970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:03.390626 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:03.390593    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:05.034396 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:05.034353    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:10:05.034396 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:05.034412    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:10:05.046961 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:05.046934    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:10:05.679435 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:05.679405    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:10:10.999671 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:10.999631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:11.154476 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:11.154435    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:11.780367 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:11.780336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:18.588327 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:18.588290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:18.740985 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:18.740951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:18.879207 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:18.879136    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:21.340331 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:21.340296    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:10:29.457110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:29.457077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:29.680895 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:29.680851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:40.636101 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:40.636052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:46.184279 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:46.184239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:50.945251 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:50.945217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:56.579525 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:56.579471    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:59.642860 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:59.642828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:10:59.790263 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:10:59.790228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:00.075649 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:00.075558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:00.680377 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:00.680344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:01.049428 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:01.049351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:03.389209 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:03.389177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:03.988006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:03.987975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:04.328610 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:04.328533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:05.385154 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:05.385117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:07.220974 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:07.220939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:09.146479 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:09.146441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:10.622613 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:10.622574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:11.773123 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:11.773083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:12.968405 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:12.968362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:14.401148 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:14.401113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:14.538798 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:14.538758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:14.654425 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:14.654360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:15.242565 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.242535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:15.356349 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.356319    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:15.528739 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.528663    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"]
Sep 18 19:11:15.532548 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.532527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:15.535999 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.535970    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1d6196\""
Sep 18 19:11:15.535999 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.535970    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1d6196-dockercfg-mhvjx\""
Sep 18 19:11:15.550470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.550441    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"]
Sep 18 19:11:15.588263 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.588242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnr9\" (UniqueName: \"kubernetes.io/projected/70ef8942-d724-4eb9-9c5e-b6266181f621-kube-api-access-slnr9\") pod \"payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:15.588364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.588285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ef8942-d724-4eb9-9c5e-b6266181f621-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:15.688861 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.688834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnr9\" (UniqueName: \"kubernetes.io/projected/70ef8942-d724-4eb9-9c5e-b6266181f621-kube-api-access-slnr9\") pod \"payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:15.689030 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.688899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ef8942-d724-4eb9-9c5e-b6266181f621-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:15.689539 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.689515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ef8942-d724-4eb9-9c5e-b6266181f621-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:15.698178 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.698154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnr9\" (UniqueName: \"kubernetes.io/projected/70ef8942-d724-4eb9-9c5e-b6266181f621-kube-api-access-slnr9\") pod \"payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:15.842393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:15.842336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:16.129642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:16.129612    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"]
Sep 18 19:11:16.130786 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:11:16.130759    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ef8942_d724_4eb9_9c5e_b6266181f621.slice/crio-8fb76645576e0efafa9ffaeb31010f6659f62aeda3790ca85126f230cdb8d33c WatchSource:0}: Error finding container 8fb76645576e0efafa9ffaeb31010f6659f62aeda3790ca85126f230cdb8d33c: Status 404 returned error can't find the container with id 8fb76645576e0efafa9ffaeb31010f6659f62aeda3790ca85126f230cdb8d33c
Sep 18 19:11:17.079110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:17.079054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr" event={"ID":"70ef8942-d724-4eb9-9c5e-b6266181f621","Type":"ContainerStarted","Data":"f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc"}
Sep 18 19:11:17.079500 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:17.079119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr" event={"ID":"70ef8942-d724-4eb9-9c5e-b6266181f621","Type":"ContainerStarted","Data":"8fb76645576e0efafa9ffaeb31010f6659f62aeda3790ca85126f230cdb8d33c"}
Sep 18 19:11:17.079500 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:17.079149    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:17.109504 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:17.109454    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr" podStartSLOduration=2.109439461 podStartE2EDuration="2.109439461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:11:17.107383788 +0000 UTC m=+1600.900060738" watchObservedRunningTime="2026-09-18 19:11:17.109439461 +0000 UTC m=+1600.902116391"
Sep 18 19:11:17.183521 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:17.183495    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:19.198113 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:19.197996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:28.088075 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:28.088040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:11:28.696530 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:28.696498    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:11:36.720832 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:36.720795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:37.092767 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:37.092684    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:38.395834 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:38.395795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:39.195939 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:39.195909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:39.745683 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:39.745645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:40.330314 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:40.330280    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:40.647170 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:40.647138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:41.735655 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:41.735616    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"]
Sep 18 19:11:41.754844 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:41.754814    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"]
Sep 18 19:11:41.873971 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:41.873949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:42.055257 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:42.055170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:43.523842 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:43.523792    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:43.900065 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:43.900037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:44.134048 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:44.134019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:45.305604 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:45.305574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:47.921262 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:47.921227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:49.283502 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:49.283467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:50.852017 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:50.851980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:52.226615 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.226573    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"]
Sep 18 19:11:52.230827 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.230800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.233739 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.233719    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-60d8bd-serving-cert\""
Sep 18 19:11:52.256807 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.256773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"]
Sep 18 19:11:52.308787 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.308750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.309003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.308835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-metrics-certs\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.309003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.308955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnhz\" (UniqueName: \"kubernetes.io/projected/661bb2ae-4ac2-453f-b768-c2bf290b0c29-kube-api-access-hrnhz\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.309003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.308992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661bb2ae-4ac2-453f-b768-c2bf290b0c29-openshift-service-ca\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.378321 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.378279    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:52.410383 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.410347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-metrics-certs\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.410555 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.410444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnhz\" (UniqueName: \"kubernetes.io/projected/661bb2ae-4ac2-453f-b768-c2bf290b0c29-kube-api-access-hrnhz\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.410555 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.410481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661bb2ae-4ac2-453f-b768-c2bf290b0c29-openshift-service-ca\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.410555 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.410544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.410684 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:11:52.410639    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-serving-cert: secret "maas-api-e2e-api-b-60d8bd-serving-cert" not found
Sep 18 19:11:52.410740 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:11:52.410725    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls podName:661bb2ae-4ac2-453f-b768-c2bf290b0c29 nodeName:}" failed. No retries permitted until 2026-09-18 19:11:52.910700962 +0000 UTC m=+1636.703377872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls") pod "maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" (UID: "661bb2ae-4ac2-453f-b768-c2bf290b0c29") : secret "maas-api-e2e-api-b-60d8bd-serving-cert" not found
Sep 18 19:11:52.411318 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.411230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661bb2ae-4ac2-453f-b768-c2bf290b0c29-openshift-service-ca\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.413469 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.413442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-metrics-certs\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.438251 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.438218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnhz\" (UniqueName: \"kubernetes.io/projected/661bb2ae-4ac2-453f-b768-c2bf290b0c29-kube-api-access-hrnhz\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.555537 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.555439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:52.699604 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.699574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:52.916950 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.916913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:52.919556 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:52.919529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls\") pod \"maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:53.142310 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:53.142277    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:53.235576 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:53.235539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:53.344066 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:53.344031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:53.456082 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:53.456051    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"]
Sep 18 19:11:53.458224 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:11:53.458195    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661bb2ae_4ac2_453f_b768_c2bf290b0c29.slice/crio-b99baccdcbc66187dc79226821ce4d9a66165b70d0fe566d1e684111e122d4b9 WatchSource:0}: Error finding container b99baccdcbc66187dc79226821ce4d9a66165b70d0fe566d1e684111e122d4b9: Status 404 returned error can't find the container with id b99baccdcbc66187dc79226821ce4d9a66165b70d0fe566d1e684111e122d4b9
Sep 18 19:11:54.088275 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:54.088239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:54.297107 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:54.297058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" event={"ID":"661bb2ae-4ac2-453f-b768-c2bf290b0c29","Type":"ContainerStarted","Data":"3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9"}
Sep 18 19:11:54.297624 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:54.297122    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" event={"ID":"661bb2ae-4ac2-453f-b768-c2bf290b0c29","Type":"ContainerStarted","Data":"b99baccdcbc66187dc79226821ce4d9a66165b70d0fe566d1e684111e122d4b9"}
Sep 18 19:11:54.297624 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:54.297228    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:11:54.320012 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:54.319937    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" podStartSLOduration=1.9070262580000001 podStartE2EDuration="2.319866906s" totalImagesPullingTime="412.840648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:11:54.31938839 +0000 UTC m=+1638.112065342" watchObservedRunningTime="2026-09-18 19:11:54.319866906 +0000 UTC m=+1638.112543838"
Sep 18 19:11:54.950812 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:54.950760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:56.106064 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:56.106027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:56.756546 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:56.756506    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8" podUID="adcde6df-f011-4e08-95a3-864c9cd8f0a3" containerName="main" containerID="cri-o://a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a" gracePeriod=45
Sep 18 19:11:56.777274 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:56.777245    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp" podUID="60e0c692-8dbb-4645-82ca-70ae0429e810" containerName="main" containerID="cri-o://1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020" gracePeriod=45
Sep 18 19:11:57.049236 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.049211    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:11:57.052747 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.052729    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:11:57.156875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.156769    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tmp-dir\") pod \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") "
Sep 18 19:11:57.156875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.156819    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e0c692-8dbb-4645-82ca-70ae0429e810-tls-certs\") pod \"60e0c692-8dbb-4645-82ca-70ae0429e810\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") "
Sep 18 19:11:57.156875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.156853    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-home\") pod \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") "
Sep 18 19:11:57.156875 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.156920    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tls-certs\") pod \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.156941    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhtmv\" (UniqueName: \"kubernetes.io/projected/adcde6df-f011-4e08-95a3-864c9cd8f0a3-kube-api-access-lhtmv\") pod \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.156963    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-dshm\") pod \"60e0c692-8dbb-4645-82ca-70ae0429e810\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.156983    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-model-cache\") pod \"60e0c692-8dbb-4645-82ca-70ae0429e810\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-home\") pod \"60e0c692-8dbb-4645-82ca-70ae0429e810\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157053    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-dshm\") pod \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157082    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-tmp-dir\") pod \"60e0c692-8dbb-4645-82ca-70ae0429e810\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157077    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adcde6df-f011-4e08-95a3-864c9cd8f0a3" (UID: "adcde6df-f011-4e08-95a3-864c9cd8f0a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2h9g\" (UniqueName: \"kubernetes.io/projected/60e0c692-8dbb-4645-82ca-70ae0429e810-kube-api-access-d2h9g\") pod \"60e0c692-8dbb-4645-82ca-70ae0429e810\" (UID: \"60e0c692-8dbb-4645-82ca-70ae0429e810\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157169    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-model-cache\") pod \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\" (UID: \"adcde6df-f011-4e08-95a3-864c9cd8f0a3\") "
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157219    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-home" (OuterVolumeSpecName: "home") pod "adcde6df-f011-4e08-95a3-864c9cd8f0a3" (UID: "adcde6df-f011-4e08-95a3-864c9cd8f0a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-model-cache" (OuterVolumeSpecName: "model-cache") pod "60e0c692-8dbb-4645-82ca-70ae0429e810" (UID: "60e0c692-8dbb-4645-82ca-70ae0429e810"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.157594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157502    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "adcde6df-f011-4e08-95a3-864c9cd8f0a3" (UID: "adcde6df-f011-4e08-95a3-864c9cd8f0a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.158364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157635    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.158364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157654    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.158364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157667    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.158364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157679    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.158364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.157862    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60e0c692-8dbb-4645-82ca-70ae0429e810" (UID: "60e0c692-8dbb-4645-82ca-70ae0429e810"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.158364 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.158110    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-home" (OuterVolumeSpecName: "home") pod "60e0c692-8dbb-4645-82ca-70ae0429e810" (UID: "60e0c692-8dbb-4645-82ca-70ae0429e810"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.160352 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.160313    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-dshm" (OuterVolumeSpecName: "dshm") pod "adcde6df-f011-4e08-95a3-864c9cd8f0a3" (UID: "adcde6df-f011-4e08-95a3-864c9cd8f0a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.160496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.160382    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adcde6df-f011-4e08-95a3-864c9cd8f0a3" (UID: "adcde6df-f011-4e08-95a3-864c9cd8f0a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:11:57.160496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.160480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcde6df-f011-4e08-95a3-864c9cd8f0a3-kube-api-access-lhtmv" (OuterVolumeSpecName: "kube-api-access-lhtmv") pod "adcde6df-f011-4e08-95a3-864c9cd8f0a3" (UID: "adcde6df-f011-4e08-95a3-864c9cd8f0a3"). InnerVolumeSpecName "kube-api-access-lhtmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:11:57.160637 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.160489    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e0c692-8dbb-4645-82ca-70ae0429e810-kube-api-access-d2h9g" (OuterVolumeSpecName: "kube-api-access-d2h9g") pod "60e0c692-8dbb-4645-82ca-70ae0429e810" (UID: "60e0c692-8dbb-4645-82ca-70ae0429e810"). InnerVolumeSpecName "kube-api-access-d2h9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:11:57.160637 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.160608    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e0c692-8dbb-4645-82ca-70ae0429e810-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60e0c692-8dbb-4645-82ca-70ae0429e810" (UID: "60e0c692-8dbb-4645-82ca-70ae0429e810"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:11:57.161069 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.161040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-dshm" (OuterVolumeSpecName: "dshm") pod "60e0c692-8dbb-4645-82ca-70ae0429e810" (UID: "60e0c692-8dbb-4645-82ca-70ae0429e810"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:11:57.258298 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258262    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e0c692-8dbb-4645-82ca-70ae0429e810-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.258298 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258293    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adcde6df-f011-4e08-95a3-864c9cd8f0a3-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.258298 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258301    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhtmv\" (UniqueName: \"kubernetes.io/projected/adcde6df-f011-4e08-95a3-864c9cd8f0a3-kube-api-access-lhtmv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.258523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258310    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.258523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258318    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.258523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258327    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adcde6df-f011-4e08-95a3-864c9cd8f0a3-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.258523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258334    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e0c692-8dbb-4645-82ca-70ae0429e810-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.258523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.258344    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2h9g\" (UniqueName: \"kubernetes.io/projected/60e0c692-8dbb-4645-82ca-70ae0429e810-kube-api-access-d2h9g\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:57.274175 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.274098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:57.315070 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.315042    2582 generic.go:356] "Generic (PLEG): container finished" podID="adcde6df-f011-4e08-95a3-864c9cd8f0a3" containerID="a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a" exitCode=0
Sep 18 19:11:57.315159 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.315082    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"
Sep 18 19:11:57.315159 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.315134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8" event={"ID":"adcde6df-f011-4e08-95a3-864c9cd8f0a3","Type":"ContainerDied","Data":"a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a"}
Sep 18 19:11:57.315249 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.315174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8" event={"ID":"adcde6df-f011-4e08-95a3-864c9cd8f0a3","Type":"ContainerDied","Data":"7a279663fc4669cac796d517f829bef140db9ce5d820dd4fd55f6dc76a414d34"}
Sep 18 19:11:57.315249 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.315198    2582 scope.go:117] "RemoveContainer" containerID="a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a"
Sep 18 19:11:57.316924 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.316875    2582 generic.go:356] "Generic (PLEG): container finished" podID="60e0c692-8dbb-4645-82ca-70ae0429e810" containerID="1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020" exitCode=0
Sep 18 19:11:57.317046 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.316933    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"
Sep 18 19:11:57.317046 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.316935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp" event={"ID":"60e0c692-8dbb-4645-82ca-70ae0429e810","Type":"ContainerDied","Data":"1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020"}
Sep 18 19:11:57.317171 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.317049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp" event={"ID":"60e0c692-8dbb-4645-82ca-70ae0429e810","Type":"ContainerDied","Data":"9d2103e7aab7f9a979971b711cf947737565a9d380a0825ee8cd1774dd2bd717"}
Sep 18 19:11:57.325683 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.325536    2582 scope.go:117] "RemoveContainer" containerID="a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a"
Sep 18 19:11:57.325863 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:11:57.325836    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a\": container with ID starting with a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a not found: ID does not exist" containerID="a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a"
Sep 18 19:11:57.326039 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.326006    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a"} err="failed to get container status \"a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a\": rpc error: code = NotFound desc = could not find container \"a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a\": container with ID starting with a8264e01ac0ad544f514c663a4fcbaecc95c10363b9845518315c90d4671ab2a not found: ID does not exist"
Sep 18 19:11:57.326127 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.326038    2582 scope.go:117] "RemoveContainer" containerID="1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020"
Sep 18 19:11:57.334764 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.334746    2582 scope.go:117] "RemoveContainer" containerID="1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020"
Sep 18 19:11:57.335076 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:11:57.335059    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020\": container with ID starting with 1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020 not found: ID does not exist" containerID="1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020"
Sep 18 19:11:57.335125 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.335082    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020"} err="failed to get container status \"1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020\": rpc error: code = NotFound desc = could not find container \"1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020\": container with ID starting with 1d72d5841536e781a954b937e74d60e39d4c488cb4f492173558262790b7d020 not found: ID does not exist"
Sep 18 19:11:57.361925 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.361877    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"]
Sep 18 19:11:57.366493 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.366456    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/simulator-w0-1a444c-kserve-57cb6f879b-h27wp"]
Sep 18 19:11:57.384219 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.384187    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"]
Sep 18 19:11:57.388933 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.388913    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a444c/premium-w0-1a444c-kserve-7448b6ddd5-vlhf8"]
Sep 18 19:11:57.394455 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:57.394437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:58.118756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.118673    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"]
Sep 18 19:11:58.118948 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.118923    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt" podUID="74283389-8b96-40fd-b345-a91801ec8261" containerName="maas-api" containerID="cri-o://f110125b39c93a4cdbcff438db4d53892ebdab5da9037abd0591817d91a9aff7" gracePeriod=30
Sep 18 19:11:58.164511 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.164486    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:58.170006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.169981    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"]
Sep 18 19:11:58.174179 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.174158    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a444c-29829300-frxjm"]
Sep 18 19:11:58.204900 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.204854    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"]
Sep 18 19:11:58.205278 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.205243    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn" podUID="32ebcb1a-c7a9-4c94-a87e-2453a042765e" containerName="payload-processing" containerID="cri-o://c95cf18d77ab809ec0ae78a7a61c7cb84b1800884304b60fbad672569e022e71" gracePeriod=30
Sep 18 19:11:58.271949 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.271916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:58.326191 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.326159    2582 generic.go:356] "Generic (PLEG): container finished" podID="32ebcb1a-c7a9-4c94-a87e-2453a042765e" containerID="c95cf18d77ab809ec0ae78a7a61c7cb84b1800884304b60fbad672569e022e71" exitCode=0
Sep 18 19:11:58.326272 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.326228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn" event={"ID":"32ebcb1a-c7a9-4c94-a87e-2453a042765e","Type":"ContainerDied","Data":"c95cf18d77ab809ec0ae78a7a61c7cb84b1800884304b60fbad672569e022e71"}
Sep 18 19:11:58.329164 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.329120    2582 generic.go:356] "Generic (PLEG): container finished" podID="74283389-8b96-40fd-b345-a91801ec8261" containerID="f110125b39c93a4cdbcff438db4d53892ebdab5da9037abd0591817d91a9aff7" exitCode=0
Sep 18 19:11:58.338208 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.338010    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e0c692-8dbb-4645-82ca-70ae0429e810" path="/var/lib/kubelet/pods/60e0c692-8dbb-4645-82ca-70ae0429e810/volumes"
Sep 18 19:11:58.338645 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.338622    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adcde6df-f011-4e08-95a3-864c9cd8f0a3" path="/var/lib/kubelet/pods/adcde6df-f011-4e08-95a3-864c9cd8f0a3/volumes"
Sep 18 19:11:58.339089 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.339070    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b290be6f-2408-4625-ae33-70277d380e3d" path="/var/lib/kubelet/pods/b290be6f-2408-4625-ae33-70277d380e3d/volumes"
Sep 18 19:11:58.339549 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.339524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt" event={"ID":"74283389-8b96-40fd-b345-a91801ec8261","Type":"ContainerDied","Data":"f110125b39c93a4cdbcff438db4d53892ebdab5da9037abd0591817d91a9aff7"}
Sep 18 19:11:58.378034 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.378007    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:58.457241 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.457217    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 19:11:58.478835 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.478812    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 19:11:58.521546 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.521488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:58.570051 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570022    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf4zm\" (UniqueName: \"kubernetes.io/projected/74283389-8b96-40fd-b345-a91801ec8261-kube-api-access-sf4zm\") pod \"74283389-8b96-40fd-b345-a91801ec8261\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") "
Sep 18 19:11:58.570207 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570061    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74283389-8b96-40fd-b345-a91801ec8261-openshift-service-ca\") pod \"74283389-8b96-40fd-b345-a91801ec8261\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") "
Sep 18 19:11:58.570207 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570126    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-metrics-certs\") pod \"74283389-8b96-40fd-b345-a91801ec8261\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") "
Sep 18 19:11:58.570207 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570182    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ebcb1a-c7a9-4c94-a87e-2453a042765e-plugins-config-volume\") pod \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") "
Sep 18 19:11:58.570387 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570236    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-maas-api-tls\") pod \"74283389-8b96-40fd-b345-a91801ec8261\" (UID: \"74283389-8b96-40fd-b345-a91801ec8261\") "
Sep 18 19:11:58.570387 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570278    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zffn\" (UniqueName: \"kubernetes.io/projected/32ebcb1a-c7a9-4c94-a87e-2453a042765e-kube-api-access-6zffn\") pod \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\" (UID: \"32ebcb1a-c7a9-4c94-a87e-2453a042765e\") "
Sep 18 19:11:58.570517 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570489    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74283389-8b96-40fd-b345-a91801ec8261-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74283389-8b96-40fd-b345-a91801ec8261" (UID: "74283389-8b96-40fd-b345-a91801ec8261"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:11:58.570604 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ebcb1a-c7a9-4c94-a87e-2453a042765e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32ebcb1a-c7a9-4c94-a87e-2453a042765e" (UID: "32ebcb1a-c7a9-4c94-a87e-2453a042765e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:11:58.570680 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.570656    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74283389-8b96-40fd-b345-a91801ec8261-openshift-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:58.572606 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.572577    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74283389-8b96-40fd-b345-a91801ec8261" (UID: "74283389-8b96-40fd-b345-a91801ec8261"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:11:58.572606 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.572593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ebcb1a-c7a9-4c94-a87e-2453a042765e-kube-api-access-6zffn" (OuterVolumeSpecName: "kube-api-access-6zffn") pod "32ebcb1a-c7a9-4c94-a87e-2453a042765e" (UID: "32ebcb1a-c7a9-4c94-a87e-2453a042765e"). InnerVolumeSpecName "kube-api-access-6zffn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:11:58.572908 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.572867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "74283389-8b96-40fd-b345-a91801ec8261" (UID: "74283389-8b96-40fd-b345-a91801ec8261"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:11:58.572908 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.572892    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74283389-8b96-40fd-b345-a91801ec8261-kube-api-access-sf4zm" (OuterVolumeSpecName: "kube-api-access-sf4zm") pod "74283389-8b96-40fd-b345-a91801ec8261" (UID: "74283389-8b96-40fd-b345-a91801ec8261"). InnerVolumeSpecName "kube-api-access-sf4zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:11:58.671934 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.671876    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf4zm\" (UniqueName: \"kubernetes.io/projected/74283389-8b96-40fd-b345-a91801ec8261-kube-api-access-sf4zm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:58.671934 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.671931    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-metrics-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:58.672118 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.671941    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32ebcb1a-c7a9-4c94-a87e-2453a042765e-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:58.672118 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.671950    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74283389-8b96-40fd-b345-a91801ec8261-maas-api-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:58.672118 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.671960    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zffn\" (UniqueName: \"kubernetes.io/projected/32ebcb1a-c7a9-4c94-a87e-2453a042765e-kube-api-access-6zffn\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:11:58.849591 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:58.849550    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:59.253442 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.253394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:11:59.335822 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.335798    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"
Sep 18 19:11:59.336007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.335844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt" event={"ID":"74283389-8b96-40fd-b345-a91801ec8261","Type":"ContainerDied","Data":"ea04f5c22f3ce0767f1c572b854757c64127fb51622c6e3caf1138ce6939a8a1"}
Sep 18 19:11:59.336007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.335909    2582 scope.go:117] "RemoveContainer" containerID="f110125b39c93a4cdbcff438db4d53892ebdab5da9037abd0591817d91a9aff7"
Sep 18 19:11:59.337387 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.337320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn" event={"ID":"32ebcb1a-c7a9-4c94-a87e-2453a042765e","Type":"ContainerDied","Data":"f24bb4062d4fd201cc913a53ae84b12a5c278516bf889151e7ed0a807d31edf0"}
Sep 18 19:11:59.337387 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.337332    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"
Sep 18 19:11:59.345227 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.345209    2582 scope.go:117] "RemoveContainer" containerID="c95cf18d77ab809ec0ae78a7a61c7cb84b1800884304b60fbad672569e022e71"
Sep 18 19:11:59.372491 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.372469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"]
Sep 18 19:11:59.381279 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.381255    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1a444c-5455b8549c-55shn"]
Sep 18 19:11:59.403761 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.403740    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"]
Sep 18 19:11:59.407297 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:11:59.407278    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a444c-5b9f789fbc-4hhqt"]
Sep 18 19:12:00.309246 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:00.309215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:12:00.335624 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:00.335595    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ebcb1a-c7a9-4c94-a87e-2453a042765e" path="/var/lib/kubelet/pods/32ebcb1a-c7a9-4c94-a87e-2453a042765e/volumes"
Sep 18 19:12:00.336038 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:00.336025    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74283389-8b96-40fd-b345-a91801ec8261" path="/var/lib/kubelet/pods/74283389-8b96-40fd-b345-a91801ec8261/volumes"
Sep 18 19:12:00.457232 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:00.457197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:01.725443 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:01.725393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:02.583587 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:02.583545    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:03.663327 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:03.663291    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:04.022761 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:04.022669    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:12:04.626563 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:04.626533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:05.689552 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:05.689522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:06.775398 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:06.775365    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:06.978596 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:06.978563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:07.190032 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:07.189994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:08.722406 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:08.722358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:08.875511 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:08.875473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:09.970229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:09.970192    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:11.530070 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:11.530019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:12.710091 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:12.710057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:13.799507 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:13.799466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:14.680057 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:14.680015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:15.561804 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:15.561750    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:16.630341 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:16.630309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:17.754590 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:17.754526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:18.620055 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:18.620012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:18.725215 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:18.725182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:19.356512 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:19.356479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:20.788395 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:20.788361    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:22.097927 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:22.097867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:23.946732 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:23.946683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:25.167423 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:25.167383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:28.033756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.033705    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:28.069654 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.069612    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"]
Sep 18 19:12:28.069994 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.069929    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr" podUID="70ef8942-d724-4eb9-9c5e-b6266181f621" containerName="payload-processing" containerID="cri-o://f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc" gracePeriod=30
Sep 18 19:12:28.087188 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.087139    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr" podUID="70ef8942-d724-4eb9-9c5e-b6266181f621" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.74:9004: connect: connection refused"
Sep 18 19:12:28.132503 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.132467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:28.232835 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.232801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:28.334580 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.334551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:28.340262 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.340236    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:12:28.373142 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.373099    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slnr9\" (UniqueName: \"kubernetes.io/projected/70ef8942-d724-4eb9-9c5e-b6266181f621-kube-api-access-slnr9\") pod \"70ef8942-d724-4eb9-9c5e-b6266181f621\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") "
Sep 18 19:12:28.373341 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.373171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ef8942-d724-4eb9-9c5e-b6266181f621-plugins-config-volume\") pod \"70ef8942-d724-4eb9-9c5e-b6266181f621\" (UID: \"70ef8942-d724-4eb9-9c5e-b6266181f621\") "
Sep 18 19:12:28.373920 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.373864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ef8942-d724-4eb9-9c5e-b6266181f621-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70ef8942-d724-4eb9-9c5e-b6266181f621" (UID: "70ef8942-d724-4eb9-9c5e-b6266181f621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:12:28.375740 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.375707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ef8942-d724-4eb9-9c5e-b6266181f621-kube-api-access-slnr9" (OuterVolumeSpecName: "kube-api-access-slnr9") pod "70ef8942-d724-4eb9-9c5e-b6266181f621" (UID: "70ef8942-d724-4eb9-9c5e-b6266181f621"). InnerVolumeSpecName "kube-api-access-slnr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:12:28.474869 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.474824    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slnr9\" (UniqueName: \"kubernetes.io/projected/70ef8942-d724-4eb9-9c5e-b6266181f621-kube-api-access-slnr9\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:12:28.474869 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.474859    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ef8942-d724-4eb9-9c5e-b6266181f621-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:12:28.507281 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.507242    2582 generic.go:356] "Generic (PLEG): container finished" podID="70ef8942-d724-4eb9-9c5e-b6266181f621" containerID="f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc" exitCode=0
Sep 18 19:12:28.507444 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.507296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr" event={"ID":"70ef8942-d724-4eb9-9c5e-b6266181f621","Type":"ContainerDied","Data":"f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc"}
Sep 18 19:12:28.507444 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.507298    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"
Sep 18 19:12:28.507444 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.507339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr" event={"ID":"70ef8942-d724-4eb9-9c5e-b6266181f621","Type":"ContainerDied","Data":"8fb76645576e0efafa9ffaeb31010f6659f62aeda3790ca85126f230cdb8d33c"}
Sep 18 19:12:28.507444 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.507362    2582 scope.go:117] "RemoveContainer" containerID="f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc"
Sep 18 19:12:28.517802 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.517778    2582 scope.go:117] "RemoveContainer" containerID="f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc"
Sep 18 19:12:28.518189 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:12:28.518167    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc\": container with ID starting with f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc not found: ID does not exist" containerID="f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc"
Sep 18 19:12:28.518268 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.518199    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc"} err="failed to get container status \"f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc\": rpc error: code = NotFound desc = could not find container \"f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc\": container with ID starting with f305a265d2f83d373947515f3dfa48f929c336ea26c0d3f65ee6368b8d85ffdc not found: ID does not exist"
Sep 18 19:12:28.530350 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.530322    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"]
Sep 18 19:12:28.535165 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.535143    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d6196-6dcb449977-fglkr"]
Sep 18 19:12:28.704243 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.704211    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:28.808461 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:28.808423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:29.817746 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:29.817700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:30.336775 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:30.336738    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ef8942-d724-4eb9-9c5e-b6266181f621" path="/var/lib/kubelet/pods/70ef8942-d724-4eb9-9c5e-b6266181f621/volumes"
Sep 18 19:12:31.081951 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:31.081915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:32.678166 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:32.678134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:33.970554 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:33.970519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:34.180215 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:34.180178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:35.251057 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:35.251022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:36.531321 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:36.531287    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:38.521601 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:38.521566    2582 scope.go:117] "RemoveContainer" containerID="d6a5cd110d06f0bb6366956e6acf69cca6feb0e30d3262250709ea1ebc83a521"
Sep 18 19:12:38.681618 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:38.681584    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:38.828959 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:38.828854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:39.043434 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:39.043399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:40.185285 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:40.185234    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:12:40.564028 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:40.563928    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:41.552643 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:41.552607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:42.364896 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:42.364859    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:42.868368 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:42.868335    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:43.943330 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:43.943295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:45.095245 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:45.095207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:46.078780 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:46.078748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:47.203576 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:47.203541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:55.105134 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.105031    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"]
Sep 18 19:12:55.105716 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.105321    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" podUID="661bb2ae-4ac2-453f-b768-c2bf290b0c29" containerName="maas-api" containerID="cri-o://3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9" gracePeriod=30
Sep 18 19:12:55.146612 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.146571    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:55.248518 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.248478    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:55.345721 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.345602    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:55.363260 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.363189    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:12:55.469458 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.469424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:55.546813 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.546753    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-metrics-certs\") pod \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") "
Sep 18 19:12:55.547052 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.546832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661bb2ae-4ac2-453f-b768-c2bf290b0c29-openshift-service-ca\") pod \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") "
Sep 18 19:12:55.547052 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.546910    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls\") pod \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") "
Sep 18 19:12:55.547052 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.546956    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrnhz\" (UniqueName: \"kubernetes.io/projected/661bb2ae-4ac2-453f-b768-c2bf290b0c29-kube-api-access-hrnhz\") pod \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\" (UID: \"661bb2ae-4ac2-453f-b768-c2bf290b0c29\") "
Sep 18 19:12:55.547266 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.547235    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661bb2ae-4ac2-453f-b768-c2bf290b0c29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "661bb2ae-4ac2-453f-b768-c2bf290b0c29" (UID: "661bb2ae-4ac2-453f-b768-c2bf290b0c29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:12:55.549227 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.549200    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "661bb2ae-4ac2-453f-b768-c2bf290b0c29" (UID: "661bb2ae-4ac2-453f-b768-c2bf290b0c29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:12:55.549342 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.549286    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661bb2ae-4ac2-453f-b768-c2bf290b0c29-kube-api-access-hrnhz" (OuterVolumeSpecName: "kube-api-access-hrnhz") pod "661bb2ae-4ac2-453f-b768-c2bf290b0c29" (UID: "661bb2ae-4ac2-453f-b768-c2bf290b0c29"). InnerVolumeSpecName "kube-api-access-hrnhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:12:55.549342 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.549290    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "661bb2ae-4ac2-453f-b768-c2bf290b0c29" (UID: "661bb2ae-4ac2-453f-b768-c2bf290b0c29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:12:55.647751 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.647713    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661bb2ae-4ac2-453f-b768-c2bf290b0c29-openshift-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:12:55.647751 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.647748    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-maas-api-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:12:55.647751 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.647757    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnhz\" (UniqueName: \"kubernetes.io/projected/661bb2ae-4ac2-453f-b768-c2bf290b0c29-kube-api-access-hrnhz\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:12:55.648029 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.647765    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661bb2ae-4ac2-453f-b768-c2bf290b0c29-metrics-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:12:55.665895 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.665855    2582 generic.go:356] "Generic (PLEG): container finished" podID="661bb2ae-4ac2-453f-b768-c2bf290b0c29" containerID="3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9" exitCode=0
Sep 18 19:12:55.666110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.665915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" event={"ID":"661bb2ae-4ac2-453f-b768-c2bf290b0c29","Type":"ContainerDied","Data":"3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9"}
Sep 18 19:12:55.666110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.665954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" event={"ID":"661bb2ae-4ac2-453f-b768-c2bf290b0c29","Type":"ContainerDied","Data":"b99baccdcbc66187dc79226821ce4d9a66165b70d0fe566d1e684111e122d4b9"}
Sep 18 19:12:55.666110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.665974    2582 scope.go:117] "RemoveContainer" containerID="3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9"
Sep 18 19:12:55.666110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.665917    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"
Sep 18 19:12:55.676559 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.676536    2582 scope.go:117] "RemoveContainer" containerID="3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9"
Sep 18 19:12:55.676910 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:12:55.676865    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9\": container with ID starting with 3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9 not found: ID does not exist" containerID="3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9"
Sep 18 19:12:55.677027 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.676920    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9"} err="failed to get container status \"3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9\": rpc error: code = NotFound desc = could not find container \"3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9\": container with ID starting with 3682e43bd5df3c8bb2bcb8ca13f80b862b0eaf3b64145fe4dbcfc51b9ce014c9 not found: ID does not exist"
Sep 18 19:12:55.695627 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.695595    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"]
Sep 18 19:12:55.699802 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.699775    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2"]
Sep 18 19:12:55.886702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:55.886662    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:56.108657 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:56.108565    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:56.336054 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:56.336017    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="661bb2ae-4ac2-453f-b768-c2bf290b0c29" path="/var/lib/kubelet/pods/661bb2ae-4ac2-453f-b768-c2bf290b0c29/volumes"
Sep 18 19:12:57.166677 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:57.166641    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:58.303553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:58.303497    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-60d8bd-7f94985ccf-zb4m2" podUID="661bb2ae-4ac2-453f-b768-c2bf290b0c29" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.75:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:12:58.331833 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:58.331790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:59.114642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:59.114607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:12:59.817393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:12:59.817354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:01.032303 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:01.032259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:02.244486 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:02.244450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:02.983386 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:02.983321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:03.944693 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:03.944653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:04.154086 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:04.154043    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"]
Sep 18 19:13:04.222473 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:04.222373    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"]
Sep 18 19:13:04.265352 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:04.265320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:04.484566 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:04.484461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:04.714035 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:04.713999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:05.456575 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:05.456539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:06.516094 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:06.516060    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:07.328744 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:07.328700    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:13:07.524515 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:07.524477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:08.939833 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:08.939799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:10.162391 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:10.162358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:14.890368 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:14.890327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:14.961345 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:14.961310    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"]
Sep 18 19:13:14.961574 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:14.961525    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr" podUID="804e1962-ebd5-4d7b-9ea1-b7a055f69977" containerName="payload-processing" containerID="cri-o://2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4" gracePeriod=30
Sep 18 19:13:15.045849 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.045803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:15.237346 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.237318    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 19:13:15.261091 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.261051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:15.335973 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.335942    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e1962-ebd5-4d7b-9ea1-b7a055f69977-plugins-config-volume\") pod \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") "
Sep 18 19:13:15.336143 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.336001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjkwj\" (UniqueName: \"kubernetes.io/projected/804e1962-ebd5-4d7b-9ea1-b7a055f69977-kube-api-access-xjkwj\") pod \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\" (UID: \"804e1962-ebd5-4d7b-9ea1-b7a055f69977\") "
Sep 18 19:13:15.336467 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.336436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804e1962-ebd5-4d7b-9ea1-b7a055f69977-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "804e1962-ebd5-4d7b-9ea1-b7a055f69977" (UID: "804e1962-ebd5-4d7b-9ea1-b7a055f69977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:13:15.338471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.338443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804e1962-ebd5-4d7b-9ea1-b7a055f69977-kube-api-access-xjkwj" (OuterVolumeSpecName: "kube-api-access-xjkwj") pod "804e1962-ebd5-4d7b-9ea1-b7a055f69977" (UID: "804e1962-ebd5-4d7b-9ea1-b7a055f69977"). InnerVolumeSpecName "kube-api-access-xjkwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:13:15.431655 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.431611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:15.437813 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.437774    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e1962-ebd5-4d7b-9ea1-b7a055f69977-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:15.437813 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.437804    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkwj\" (UniqueName: \"kubernetes.io/projected/804e1962-ebd5-4d7b-9ea1-b7a055f69977-kube-api-access-xjkwj\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:15.664194 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.664159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:15.781737 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.781692    2582 generic.go:356] "Generic (PLEG): container finished" podID="804e1962-ebd5-4d7b-9ea1-b7a055f69977" containerID="2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4" exitCode=0
Sep 18 19:13:15.781970 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.781793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"
Sep 18 19:13:15.781970 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.781817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr" event={"ID":"804e1962-ebd5-4d7b-9ea1-b7a055f69977","Type":"ContainerDied","Data":"2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4"}
Sep 18 19:13:15.781970 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.781918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr" event={"ID":"804e1962-ebd5-4d7b-9ea1-b7a055f69977","Type":"ContainerDied","Data":"4e80e681110d092881296c65b7d1b49cb53898f131a3456b5a158c6df3c83636"}
Sep 18 19:13:15.781970 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.781953    2582 scope.go:117] "RemoveContainer" containerID="2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4"
Sep 18 19:13:15.792514 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.792486    2582 scope.go:117] "RemoveContainer" containerID="2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4"
Sep 18 19:13:15.792834 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:13:15.792811    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4\": container with ID starting with 2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4 not found: ID does not exist" containerID="2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4"
Sep 18 19:13:15.792977 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.792849    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4"} err="failed to get container status \"2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4\": rpc error: code = NotFound desc = could not find container \"2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4\": container with ID starting with 2e3847ec28644af898bf0d676b39cf9b8f036545a815069a3799f96b4e475ef4 not found: ID does not exist"
Sep 18 19:13:15.823062 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.823027    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"]
Sep 18 19:13:15.828038 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.827998    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-942883-78d6d7566c-vb2vr"]
Sep 18 19:13:15.896827 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:15.896791    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:16.202528 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:16.202494    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:16.335756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:16.335721    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="804e1962-ebd5-4d7b-9ea1-b7a055f69977" path="/var/lib/kubelet/pods/804e1962-ebd5-4d7b-9ea1-b7a055f69977/volumes"
Sep 18 19:13:16.524916 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:16.524814    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:16.711566 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:16.711520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:17.759575 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:17.759537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:18.986849 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:18.986808    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:19.174282 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.174232    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68" podUID="4ea0e6e9-ec57-405f-8b53-b698068e8edf" containerName="main" containerID="cri-o://180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be" gracePeriod=45
Sep 18 19:13:19.242954 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.242821    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x" podUID="7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" containerName="main" containerID="cri-o://eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738" gracePeriod=45
Sep 18 19:13:19.524412 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.524373    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:13:19.527212 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.527188    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 19:13:19.581286 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581248    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tmp-dir\") pod \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") "
Sep 18 19:13:19.581470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-home\") pod \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") "
Sep 18 19:13:19.581470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581353    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-model-cache\") pod \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") "
Sep 18 19:13:19.581470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581383    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqr55\" (UniqueName: \"kubernetes.io/projected/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-kube-api-access-qqr55\") pod \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") "
Sep 18 19:13:19.581470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tmp-dir\") pod \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") "
Sep 18 19:13:19.581470 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581450    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqpsb\" (UniqueName: \"kubernetes.io/projected/4ea0e6e9-ec57-405f-8b53-b698068e8edf-kube-api-access-xqpsb\") pod \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") "
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-home\") pod \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") "
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-dshm\") pod \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") "
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581556    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tls-certs\") pod \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\" (UID: \"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5\") "
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tls-certs\") pod \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") "
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581589    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" (UID: "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-home" (OuterVolumeSpecName: "home") pod "4ea0e6e9-ec57-405f-8b53-b698068e8edf" (UID: "4ea0e6e9-ec57-405f-8b53-b698068e8edf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581647    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-dshm\") pod \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") "
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" (UID: "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.581756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.581693    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-model-cache\") pod \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\" (UID: \"4ea0e6e9-ec57-405f-8b53-b698068e8edf\") "
Sep 18 19:13:19.582285 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.582082    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.582285 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.582105    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.582285 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.582120    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.582285 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.582155    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ea0e6e9-ec57-405f-8b53-b698068e8edf" (UID: "4ea0e6e9-ec57-405f-8b53-b698068e8edf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.582502 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.582352    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ea0e6e9-ec57-405f-8b53-b698068e8edf" (UID: "4ea0e6e9-ec57-405f-8b53-b698068e8edf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.582502 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.582399    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-home" (OuterVolumeSpecName: "home") pod "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" (UID: "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.587051 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.587007    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-kube-api-access-qqr55" (OuterVolumeSpecName: "kube-api-access-qqr55") pod "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" (UID: "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5"). InnerVolumeSpecName "kube-api-access-qqr55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:13:19.587325 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.587297    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-dshm" (OuterVolumeSpecName: "dshm") pod "4ea0e6e9-ec57-405f-8b53-b698068e8edf" (UID: "4ea0e6e9-ec57-405f-8b53-b698068e8edf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.587325 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.587302    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ea0e6e9-ec57-405f-8b53-b698068e8edf" (UID: "4ea0e6e9-ec57-405f-8b53-b698068e8edf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:13:19.587584 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.587562    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" (UID: "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:13:19.588134 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.588102    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea0e6e9-ec57-405f-8b53-b698068e8edf-kube-api-access-xqpsb" (OuterVolumeSpecName: "kube-api-access-xqpsb") pod "4ea0e6e9-ec57-405f-8b53-b698068e8edf" (UID: "4ea0e6e9-ec57-405f-8b53-b698068e8edf"). InnerVolumeSpecName "kube-api-access-xqpsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:13:19.588484 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.588458    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-dshm" (OuterVolumeSpecName: "dshm") pod "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" (UID: "7ec3fca7-dafe-4edf-8c2f-6290ba534ab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:13:19.683434 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683395    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqr55\" (UniqueName: \"kubernetes.io/projected/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-kube-api-access-qqr55\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683434 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683428    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683434 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683439    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqpsb\" (UniqueName: \"kubernetes.io/projected/4ea0e6e9-ec57-405f-8b53-b698068e8edf-kube-api-access-xqpsb\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683446    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683456    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683463    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683471    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea0e6e9-ec57-405f-8b53-b698068e8edf-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683479    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.683692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.683486    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea0e6e9-ec57-405f-8b53-b698068e8edf-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:19.805397 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.805309    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"
Sep 18 19:13:19.805397 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.805333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x" event={"ID":"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5","Type":"ContainerDied","Data":"eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738"}
Sep 18 19:13:19.805397 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.805379    2582 scope.go:117] "RemoveContainer" containerID="eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738"
Sep 18 19:13:19.805692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.805308    2582 generic.go:356] "Generic (PLEG): container finished" podID="7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" containerID="eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738" exitCode=0
Sep 18 19:13:19.805692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.805587    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x" event={"ID":"7ec3fca7-dafe-4edf-8c2f-6290ba534ab5","Type":"ContainerDied","Data":"9ca24df1835af8a44c296ad081ef00b8f2e8b321ca92bd55622ed7e309c8f097"}
Sep 18 19:13:19.807187 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.807161    2582 generic.go:356] "Generic (PLEG): container finished" podID="4ea0e6e9-ec57-405f-8b53-b698068e8edf" containerID="180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be" exitCode=0
Sep 18 19:13:19.807299 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.807214    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"
Sep 18 19:13:19.807299 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.807218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68" event={"ID":"4ea0e6e9-ec57-405f-8b53-b698068e8edf","Type":"ContainerDied","Data":"180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be"}
Sep 18 19:13:19.807299 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.807246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68" event={"ID":"4ea0e6e9-ec57-405f-8b53-b698068e8edf","Type":"ContainerDied","Data":"c52d4f5f3159cc883b6bcb2d7457ccc5a0633b5d3ce487f7fa23412e305a157a"}
Sep 18 19:13:19.815668 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.815648    2582 scope.go:117] "RemoveContainer" containerID="eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738"
Sep 18 19:13:19.816067 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:13:19.816039    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738\": container with ID starting with eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738 not found: ID does not exist" containerID="eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738"
Sep 18 19:13:19.816187 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.816074    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738"} err="failed to get container status \"eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738\": rpc error: code = NotFound desc = could not find container \"eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738\": container with ID starting with eb83bf99a748a63f16942d8c6abcbce9e0b0deaaae564403ab92b86da24af738 not found: ID does not exist"
Sep 18 19:13:19.816187 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.816091    2582 scope.go:117] "RemoveContainer" containerID="180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be"
Sep 18 19:13:19.824852 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.824831    2582 scope.go:117] "RemoveContainer" containerID="180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be"
Sep 18 19:13:19.825150 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:13:19.825127    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be\": container with ID starting with 180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be not found: ID does not exist" containerID="180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be"
Sep 18 19:13:19.825223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.825156    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be"} err="failed to get container status \"180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be\": rpc error: code = NotFound desc = could not find container \"180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be\": container with ID starting with 180f0708c030be6cc22e1b44e00441eecd6c74fc09c4742b2cccd5432c6e26be not found: ID does not exist"
Sep 18 19:13:19.833468 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.833443    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"]
Sep 18 19:13:19.838496 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.838471    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-942883/distinct-2-w4-942883-kserve-74b786569d-5qr68"]
Sep 18 19:13:19.855618 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.855585    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"]
Sep 18 19:13:19.861643 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:19.861612    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-942883/simulator-w4-942883-kserve-777d6c48d7-btv8x"]
Sep 18 19:13:20.068303 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:20.068205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:20.335793 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:20.335689    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea0e6e9-ec57-405f-8b53-b698068e8edf" path="/var/lib/kubelet/pods/4ea0e6e9-ec57-405f-8b53-b698068e8edf/volumes"
Sep 18 19:13:20.336260 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:20.336236    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" path="/var/lib/kubelet/pods/7ec3fca7-dafe-4edf-8c2f-6290ba534ab5/volumes"
Sep 18 19:13:21.539372 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:21.539336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:23.240311 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:23.240266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:24.655051 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:24.655021    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:24.809434 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:24.809391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:24.994810 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:24.994681    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:25.769778 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:25.769739    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:25.869908 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:25.869857    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:26.160677 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:26.160610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:26.664266 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:26.664226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:27.886065 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:27.886025    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:28.845432 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:28.845390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:29.788004 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:29.787969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:31.179729 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:31.179694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:32.201260 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:32.201223    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:13:37.109300 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.109253    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:37.151697 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.151652    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"]
Sep 18 19:13:37.152032 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.151963    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7" podUID="8515879d-6208-411e-b787-ae39b7c5e9eb" containerName="payload-pre-processing" containerID="cri-o://da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e" gracePeriod=30
Sep 18 19:13:37.205989 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.205951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:37.306503 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.306462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:37.407867 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.407833    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:13:37.449529 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.449491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:37.563856 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.563807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515879d-6208-411e-b787-ae39b7c5e9eb-plugins-config-volume\") pod \"8515879d-6208-411e-b787-ae39b7c5e9eb\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") "
Sep 18 19:13:37.564115 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.563914    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwlsx\" (UniqueName: \"kubernetes.io/projected/8515879d-6208-411e-b787-ae39b7c5e9eb-kube-api-access-fwlsx\") pod \"8515879d-6208-411e-b787-ae39b7c5e9eb\" (UID: \"8515879d-6208-411e-b787-ae39b7c5e9eb\") "
Sep 18 19:13:37.564361 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.564336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8515879d-6208-411e-b787-ae39b7c5e9eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8515879d-6208-411e-b787-ae39b7c5e9eb" (UID: "8515879d-6208-411e-b787-ae39b7c5e9eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:13:37.566374 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.566348    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8515879d-6208-411e-b787-ae39b7c5e9eb-kube-api-access-fwlsx" (OuterVolumeSpecName: "kube-api-access-fwlsx") pod "8515879d-6208-411e-b787-ae39b7c5e9eb" (UID: "8515879d-6208-411e-b787-ae39b7c5e9eb"). InnerVolumeSpecName "kube-api-access-fwlsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:13:37.665474 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.665431    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8515879d-6208-411e-b787-ae39b7c5e9eb-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:37.665474 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.665467    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlsx\" (UniqueName: \"kubernetes.io/projected/8515879d-6208-411e-b787-ae39b7c5e9eb-kube-api-access-fwlsx\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:13:37.868953 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.868915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:37.910293 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.910254    2582 generic.go:356] "Generic (PLEG): container finished" podID="8515879d-6208-411e-b787-ae39b7c5e9eb" containerID="da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e" exitCode=0
Sep 18 19:13:37.910293 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.910296    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"
Sep 18 19:13:37.910540 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.910301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7" event={"ID":"8515879d-6208-411e-b787-ae39b7c5e9eb","Type":"ContainerDied","Data":"da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e"}
Sep 18 19:13:37.910540 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.910421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7" event={"ID":"8515879d-6208-411e-b787-ae39b7c5e9eb","Type":"ContainerDied","Data":"a975c8bda71298f5b6d1ea87218cabc22ccfc9967a550d9c91d18f884ef0344e"}
Sep 18 19:13:37.910540 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.910451    2582 scope.go:117] "RemoveContainer" containerID="da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e"
Sep 18 19:13:37.920597 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.920578    2582 scope.go:117] "RemoveContainer" containerID="da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e"
Sep 18 19:13:37.920959 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:13:37.920939    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e\": container with ID starting with da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e not found: ID does not exist" containerID="da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e"
Sep 18 19:13:37.921031 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.920969    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e"} err="failed to get container status \"da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e\": rpc error: code = NotFound desc = could not find container \"da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e\": container with ID starting with da037cc4efcfab40773a284956718b3df41223eede9d811025a4bf68efb2c56e not found: ID does not exist"
Sep 18 19:13:37.962985 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.962948    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"]
Sep 18 19:13:37.972655 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:37.972609    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4805a7-657b9d9b4b-sf9g7"]
Sep 18 19:13:38.054002 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:38.053965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:38.338006 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:38.337918    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8515879d-6208-411e-b787-ae39b7c5e9eb" path="/var/lib/kubelet/pods/8515879d-6208-411e-b787-ae39b7c5e9eb/volumes"
Sep 18 19:13:39.149420 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:39.149383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:40.103454 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:40.103417    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:42.181380 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:42.181349    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:42.336346 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:42.336312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:45.236473 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:45.236426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:46.174032 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:46.173993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:46.280517 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:46.280486    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:46.517223 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:46.517124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:49.195542 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:49.195500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:13:51.517507 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:51.517473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:55.674769 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:55.674730    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"]
Sep 18 19:13:55.701204 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:55.701173    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:13:56.825128 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:56.825089    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"]
Sep 18 19:13:56.862485 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:13:56.862444    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:05.141820 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:05.141781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:14:10.695169 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:10.695105    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr" podUID="6237c2fc-8ac5-4148-b39c-60ed7c072d37" containerName="main" containerID="cri-o://8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5" gracePeriod=45
Sep 18 19:14:10.951812 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:10.951730    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:14:11.079711 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.079669    2582 generic.go:356] "Generic (PLEG): container finished" podID="6237c2fc-8ac5-4148-b39c-60ed7c072d37" containerID="8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5" exitCode=0
Sep 18 19:14:11.079711 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.079711    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"
Sep 18 19:14:11.079978 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.079760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr" event={"ID":"6237c2fc-8ac5-4148-b39c-60ed7c072d37","Type":"ContainerDied","Data":"8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5"}
Sep 18 19:14:11.079978 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.079804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr" event={"ID":"6237c2fc-8ac5-4148-b39c-60ed7c072d37","Type":"ContainerDied","Data":"58a514c64d3bfcd57a47b6d4a75e6e4207c781796442fe9b2bbce65d05bfd14b"}
Sep 18 19:14:11.079978 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.079824    2582 scope.go:117] "RemoveContainer" containerID="8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5"
Sep 18 19:14:11.084258 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-home\") pod \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") "
Sep 18 19:14:11.084398 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084292    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-model-cache\") pod \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") "
Sep 18 19:14:11.084398 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tmp-dir\") pod \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") "
Sep 18 19:14:11.084398 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084342    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tls-certs\") pod \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") "
Sep 18 19:14:11.084398 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vnqg\" (UniqueName: \"kubernetes.io/projected/6237c2fc-8ac5-4148-b39c-60ed7c072d37-kube-api-access-7vnqg\") pod \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") "
Sep 18 19:14:11.084615 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-dshm\") pod \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\" (UID: \"6237c2fc-8ac5-4148-b39c-60ed7c072d37\") "
Sep 18 19:14:11.084615 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084515    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-home" (OuterVolumeSpecName: "home") pod "6237c2fc-8ac5-4148-b39c-60ed7c072d37" (UID: "6237c2fc-8ac5-4148-b39c-60ed7c072d37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:11.084718 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084609    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6237c2fc-8ac5-4148-b39c-60ed7c072d37" (UID: "6237c2fc-8ac5-4148-b39c-60ed7c072d37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:11.084718 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084630    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-model-cache" (OuterVolumeSpecName: "model-cache") pod "6237c2fc-8ac5-4148-b39c-60ed7c072d37" (UID: "6237c2fc-8ac5-4148-b39c-60ed7c072d37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:11.084834 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084808    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:11.084930 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084833    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:11.084930 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.084846    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:11.087009 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.086983    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-dshm" (OuterVolumeSpecName: "dshm") pod "6237c2fc-8ac5-4148-b39c-60ed7c072d37" (UID: "6237c2fc-8ac5-4148-b39c-60ed7c072d37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:11.087131 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.087070    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6237c2fc-8ac5-4148-b39c-60ed7c072d37" (UID: "6237c2fc-8ac5-4148-b39c-60ed7c072d37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:14:11.087180 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.087143    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6237c2fc-8ac5-4148-b39c-60ed7c072d37-kube-api-access-7vnqg" (OuterVolumeSpecName: "kube-api-access-7vnqg") pod "6237c2fc-8ac5-4148-b39c-60ed7c072d37" (UID: "6237c2fc-8ac5-4148-b39c-60ed7c072d37"). InnerVolumeSpecName "kube-api-access-7vnqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:14:11.090716 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.090694    2582 scope.go:117] "RemoveContainer" containerID="8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5"
Sep 18 19:14:11.091104 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:14:11.091085    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5\": container with ID starting with 8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5 not found: ID does not exist" containerID="8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5"
Sep 18 19:14:11.091221 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.091113    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5"} err="failed to get container status \"8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5\": rpc error: code = NotFound desc = could not find container \"8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5\": container with ID starting with 8c1718c8923f937346f7e4addd6ab8bc9c2e97f9657aa517ee7a172867bdf5e5 not found: ID does not exist"
Sep 18 19:14:11.186413 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.186353    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6237c2fc-8ac5-4148-b39c-60ed7c072d37-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:11.186413 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.186403    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnqg\" (UniqueName: \"kubernetes.io/projected/6237c2fc-8ac5-4148-b39c-60ed7c072d37-kube-api-access-7vnqg\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:11.186413 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.186414    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6237c2fc-8ac5-4148-b39c-60ed7c072d37-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:11.255998 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.255911    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:11.419066 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.419024    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"]
Sep 18 19:14:11.427288 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.427252    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/auth-test-model-feb80d-kserve-5cc8db768c-7fbdr"]
Sep 18 19:14:11.845473 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:11.845425    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp" podUID="cb60599d-3543-458f-b8be-a9b7e6b988be" containerName="main" containerID="cri-o://003f0105a71d772d6d25c2f6ece1c4dc991f3cd144a486fa366f43ed45ef62ba" gracePeriod=45
Sep 18 19:14:12.086970 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.086940    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb60599d-3543-458f-b8be-a9b7e6b988be" containerID="003f0105a71d772d6d25c2f6ece1c4dc991f3cd144a486fa366f43ed45ef62ba" exitCode=0
Sep 18 19:14:12.087119 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.087013    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp" event={"ID":"cb60599d-3543-458f-b8be-a9b7e6b988be","Type":"ContainerDied","Data":"003f0105a71d772d6d25c2f6ece1c4dc991f3cd144a486fa366f43ed45ef62ba"}
Sep 18 19:14:12.087119 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.087060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp" event={"ID":"cb60599d-3543-458f-b8be-a9b7e6b988be","Type":"ContainerDied","Data":"aa70539a78bd4df840d8effe5afd41eeeb869b26ae2f24b53f72be16b3618507"}
Sep 18 19:14:12.087119 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.087077    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa70539a78bd4df840d8effe5afd41eeeb869b26ae2f24b53f72be16b3618507"
Sep 18 19:14:12.100211 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.100137    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:14:12.197806 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.197768    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-tmp-dir\") pod \"cb60599d-3543-458f-b8be-a9b7e6b988be\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") "
Sep 18 19:14:12.198007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.197852    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lqxk\" (UniqueName: \"kubernetes.io/projected/cb60599d-3543-458f-b8be-a9b7e6b988be-kube-api-access-2lqxk\") pod \"cb60599d-3543-458f-b8be-a9b7e6b988be\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") "
Sep 18 19:14:12.198007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.197909    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-model-cache\") pod \"cb60599d-3543-458f-b8be-a9b7e6b988be\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") "
Sep 18 19:14:12.198007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.197943    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-dshm\") pod \"cb60599d-3543-458f-b8be-a9b7e6b988be\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") "
Sep 18 19:14:12.198007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.197969    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb60599d-3543-458f-b8be-a9b7e6b988be-tls-certs\") pod \"cb60599d-3543-458f-b8be-a9b7e6b988be\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") "
Sep 18 19:14:12.198007 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.197988    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-home\") pod \"cb60599d-3543-458f-b8be-a9b7e6b988be\" (UID: \"cb60599d-3543-458f-b8be-a9b7e6b988be\") "
Sep 18 19:14:12.198243 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.198129    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb60599d-3543-458f-b8be-a9b7e6b988be" (UID: "cb60599d-3543-458f-b8be-a9b7e6b988be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:12.198342 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.198290    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb60599d-3543-458f-b8be-a9b7e6b988be" (UID: "cb60599d-3543-458f-b8be-a9b7e6b988be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:12.198342 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.198323    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:12.198499 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.198362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-home" (OuterVolumeSpecName: "home") pod "cb60599d-3543-458f-b8be-a9b7e6b988be" (UID: "cb60599d-3543-458f-b8be-a9b7e6b988be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:12.200535 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.200508    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb60599d-3543-458f-b8be-a9b7e6b988be-kube-api-access-2lqxk" (OuterVolumeSpecName: "kube-api-access-2lqxk") pod "cb60599d-3543-458f-b8be-a9b7e6b988be" (UID: "cb60599d-3543-458f-b8be-a9b7e6b988be"). InnerVolumeSpecName "kube-api-access-2lqxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:14:12.200631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.200537    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb60599d-3543-458f-b8be-a9b7e6b988be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb60599d-3543-458f-b8be-a9b7e6b988be" (UID: "cb60599d-3543-458f-b8be-a9b7e6b988be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:14:12.200631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.200560    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-dshm" (OuterVolumeSpecName: "dshm") pod "cb60599d-3543-458f-b8be-a9b7e6b988be" (UID: "cb60599d-3543-458f-b8be-a9b7e6b988be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:14:12.299391 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.299350    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:12.299391 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.299385    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb60599d-3543-458f-b8be-a9b7e6b988be-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:12.299391 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.299395    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:12.299648 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.299405    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lqxk\" (UniqueName: \"kubernetes.io/projected/cb60599d-3543-458f-b8be-a9b7e6b988be-kube-api-access-2lqxk\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:12.299648 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.299415    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb60599d-3543-458f-b8be-a9b7e6b988be-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:14:12.335642 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:12.335606    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6237c2fc-8ac5-4148-b39c-60ed7c072d37" path="/var/lib/kubelet/pods/6237c2fc-8ac5-4148-b39c-60ed7c072d37/volumes"
Sep 18 19:14:13.093946 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:13.093904    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"
Sep 18 19:14:13.116135 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:13.116090    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"]
Sep 18 19:14:13.123017 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:13.122968    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-19944d/auth-test-model-19944d-kserve-69f67c65ff-mrxrp"]
Sep 18 19:14:13.930065 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:13.930028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:14.335371 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:14.335282    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb60599d-3543-458f-b8be-a9b7e6b988be" path="/var/lib/kubelet/pods/cb60599d-3543-458f-b8be-a9b7e6b988be/volumes"
Sep 18 19:14:15.803782 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:15.803738    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:14:29.753455 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:29.753403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:14:34.099036 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:34.098988    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:14:55.494742 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:14:55.494698    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:15:08.797503 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:08.797461    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:15:12.867841 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.867801    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-b4d4e0"]
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868189    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8515879d-6208-411e-b787-ae39b7c5e9eb" containerName="payload-pre-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868201    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8515879d-6208-411e-b787-ae39b7c5e9eb" containerName="payload-pre-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868216    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="661bb2ae-4ac2-453f-b768-c2bf290b0c29" containerName="maas-api"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868221    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="661bb2ae-4ac2-453f-b768-c2bf290b0c29" containerName="maas-api"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868229    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e0c692-8dbb-4645-82ca-70ae0429e810" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868235    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e0c692-8dbb-4645-82ca-70ae0429e810" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868242    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea0e6e9-ec57-405f-8b53-b698068e8edf" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868248    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea0e6e9-ec57-405f-8b53-b698068e8edf" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868262    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74283389-8b96-40fd-b345-a91801ec8261" containerName="maas-api"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868267    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="74283389-8b96-40fd-b345-a91801ec8261" containerName="maas-api"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868273    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ebcb1a-c7a9-4c94-a87e-2453a042765e" containerName="payload-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868278    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ebcb1a-c7a9-4c94-a87e-2453a042765e" containerName="payload-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868285    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ef8942-d724-4eb9-9c5e-b6266181f621" containerName="payload-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868290    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ef8942-d724-4eb9-9c5e-b6266181f621" containerName="payload-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868296    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adcde6df-f011-4e08-95a3-864c9cd8f0a3" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868300    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="adcde6df-f011-4e08-95a3-864c9cd8f0a3" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868305    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb60599d-3543-458f-b8be-a9b7e6b988be" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868310    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb60599d-3543-458f-b8be-a9b7e6b988be" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868317    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868321    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868329    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="804e1962-ebd5-4d7b-9ea1-b7a055f69977" containerName="payload-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868334    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="804e1962-ebd5-4d7b-9ea1-b7a055f69977" containerName="payload-processing"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868339    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6237c2fc-8ac5-4148-b39c-60ed7c072d37" containerName="main"
Sep 18 19:15:12.868351 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868343    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6237c2fc-8ac5-4148-b39c-60ed7c072d37" containerName="main"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868391    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ebcb1a-c7a9-4c94-a87e-2453a042765e" containerName="payload-processing"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868399    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="804e1962-ebd5-4d7b-9ea1-b7a055f69977" containerName="payload-processing"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868405    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="74283389-8b96-40fd-b345-a91801ec8261" containerName="maas-api"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868410    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec3fca7-dafe-4edf-8c2f-6290ba534ab5" containerName="main"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868415    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ef8942-d724-4eb9-9c5e-b6266181f621" containerName="payload-processing"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868422    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb60599d-3543-458f-b8be-a9b7e6b988be" containerName="main"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868429    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6237c2fc-8ac5-4148-b39c-60ed7c072d37" containerName="main"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868434    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea0e6e9-ec57-405f-8b53-b698068e8edf" containerName="main"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868439    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="661bb2ae-4ac2-453f-b768-c2bf290b0c29" containerName="maas-api"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868445    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e0c692-8dbb-4645-82ca-70ae0429e810" containerName="main"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868449    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8515879d-6208-411e-b787-ae39b7c5e9eb" containerName="payload-pre-processing"
Sep 18 19:15:12.869109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.868454    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="adcde6df-f011-4e08-95a3-864c9cd8f0a3" containerName="main"
Sep 18 19:15:12.871447 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.871429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:12.874215 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.874190    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wgm9b\""
Sep 18 19:15:12.880794 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.880768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-b4d4e0"]
Sep 18 19:15:12.969730 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:12.969682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zbs2\" (UniqueName: \"kubernetes.io/projected/b93349aa-3f07-4290-b233-765f6b96a208-kube-api-access-9zbs2\") pod \"test-curl-7684-b4d4e0\" (UID: \"b93349aa-3f07-4290-b233-765f6b96a208\") " pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:13.070949 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:13.070903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zbs2\" (UniqueName: \"kubernetes.io/projected/b93349aa-3f07-4290-b233-765f6b96a208-kube-api-access-9zbs2\") pod \"test-curl-7684-b4d4e0\" (UID: \"b93349aa-3f07-4290-b233-765f6b96a208\") " pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:13.080471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:13.080445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zbs2\" (UniqueName: \"kubernetes.io/projected/b93349aa-3f07-4290-b233-765f6b96a208-kube-api-access-9zbs2\") pod \"test-curl-7684-b4d4e0\" (UID: \"b93349aa-3f07-4290-b233-765f6b96a208\") " pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:13.182910 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:13.182844    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:13.467992 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:13.467961    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-b4d4e0"]
Sep 18 19:15:13.470725 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:15:13.470696    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93349aa_3f07_4290_b233_765f6b96a208.slice/crio-9a4c36f4d6e1c227f62932b3f4f6e89f04f3b333a16e3a24019dd3539213a5b9 WatchSource:0}: Error finding container 9a4c36f4d6e1c227f62932b3f4f6e89f04f3b333a16e3a24019dd3539213a5b9: Status 404 returned error can't find the container with id 9a4c36f4d6e1c227f62932b3f4f6e89f04f3b333a16e3a24019dd3539213a5b9
Sep 18 19:15:14.416189 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:14.416144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-b4d4e0" event={"ID":"b93349aa-3f07-4290-b233-765f6b96a208","Type":"ContainerStarted","Data":"ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de"}
Sep 18 19:15:14.416654 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:14.416201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-b4d4e0" event={"ID":"b93349aa-3f07-4290-b233-765f6b96a208","Type":"ContainerStarted","Data":"9a4c36f4d6e1c227f62932b3f4f6e89f04f3b333a16e3a24019dd3539213a5b9"}
Sep 18 19:15:14.460772 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:14.460710    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7684-b4d4e0" podStartSLOduration=2.460689517 podStartE2EDuration="2.460689517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:14.458129011 +0000 UTC m=+1838.250805941" watchObservedRunningTime="2026-09-18 19:15:14.460689517 +0000 UTC m=+1838.253366447"
Sep 18 19:15:15.193944 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.193904    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-b4d4e0"]
Sep 18 19:15:15.196350 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.196321    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-b4d4e0"]
Sep 18 19:15:15.331639 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.331550    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-51e4d0"]
Sep 18 19:15:15.331978 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.331961    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93349aa-3f07-4290-b233-765f6b96a208" containerName="test-curl-7684-b4d4e0"
Sep 18 19:15:15.332028 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.331979    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93349aa-3f07-4290-b233-765f6b96a208" containerName="test-curl-7684-b4d4e0"
Sep 18 19:15:15.332065 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.332051    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93349aa-3f07-4290-b233-765f6b96a208" containerName="test-curl-7684-b4d4e0"
Sep 18 19:15:15.335098 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.335082    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:15.352125 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.352095    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-51e4d0"]
Sep 18 19:15:15.394056 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.394016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4x89\" (UniqueName: \"kubernetes.io/projected/ab9c4404-7497-4142-a127-9d874f29b6bb-kube-api-access-h4x89\") pod \"test-curl-7684-51e4d0\" (UID: \"ab9c4404-7497-4142-a127-9d874f29b6bb\") " pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:15.421350 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:15:15.421312    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7684-b4d4e0" container="test-curl-7684-b4d4e0"
Sep 18 19:15:15.495401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.495356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4x89\" (UniqueName: \"kubernetes.io/projected/ab9c4404-7497-4142-a127-9d874f29b6bb-kube-api-access-h4x89\") pod \"test-curl-7684-51e4d0\" (UID: \"ab9c4404-7497-4142-a127-9d874f29b6bb\") " pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:15.506258 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.506226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4x89\" (UniqueName: \"kubernetes.io/projected/ab9c4404-7497-4142-a127-9d874f29b6bb-kube-api-access-h4x89\") pod \"test-curl-7684-51e4d0\" (UID: \"ab9c4404-7497-4142-a127-9d874f29b6bb\") " pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:15.643958 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.643925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:15.919105 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:15.919073    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-51e4d0"]
Sep 18 19:15:15.921380 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:15:15.921353    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab9c4404_7497_4142_a127_9d874f29b6bb.slice/crio-b5e3e36b8743461bb01505aa5ff28242c8488b229f180cd45c659d704b49ff94 WatchSource:0}: Error finding container b5e3e36b8743461bb01505aa5ff28242c8488b229f180cd45c659d704b49ff94: Status 404 returned error can't find the container with id b5e3e36b8743461bb01505aa5ff28242c8488b229f180cd45c659d704b49ff94
Sep 18 19:15:16.427785 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:16.427743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-51e4d0" event={"ID":"ab9c4404-7497-4142-a127-9d874f29b6bb","Type":"ContainerStarted","Data":"6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf"}
Sep 18 19:15:16.428279 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:16.427798    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-51e4d0" event={"ID":"ab9c4404-7497-4142-a127-9d874f29b6bb","Type":"ContainerStarted","Data":"b5e3e36b8743461bb01505aa5ff28242c8488b229f180cd45c659d704b49ff94"}
Sep 18 19:15:16.428279 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:16.427953    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7684-b4d4e0" podUID="b93349aa-3f07-4290-b233-765f6b96a208" containerName="test-curl-7684-b4d4e0" containerID="cri-o://ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de" gracePeriod=2
Sep 18 19:15:16.452786 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:16.452724    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7684-51e4d0" podStartSLOduration=1.452706936 podStartE2EDuration="1.452706936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:16.45030348 +0000 UTC m=+1840.242980402" watchObservedRunningTime="2026-09-18 19:15:16.452706936 +0000 UTC m=+1840.245383867"
Sep 18 19:15:17.132499 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:17.132458    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-51e4d0"]
Sep 18 19:15:17.138058 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:17.137970    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-51e4d0"]
Sep 18 19:15:17.432843 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:15:17.432811    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7684-51e4d0" container="test-curl-7684-51e4d0"
Sep 18 19:15:18.438195 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:18.438159    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7684-51e4d0" podUID="ab9c4404-7497-4142-a127-9d874f29b6bb" containerName="test-curl-7684-51e4d0" containerID="cri-o://6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf" gracePeriod=2
Sep 18 19:15:18.620549 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:18.620518    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:18.729526 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:18.729419    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zbs2\" (UniqueName: \"kubernetes.io/projected/b93349aa-3f07-4290-b233-765f6b96a208-kube-api-access-9zbs2\") pod \"b93349aa-3f07-4290-b233-765f6b96a208\" (UID: \"b93349aa-3f07-4290-b233-765f6b96a208\") "
Sep 18 19:15:18.731785 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:18.731757    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93349aa-3f07-4290-b233-765f6b96a208-kube-api-access-9zbs2" (OuterVolumeSpecName: "kube-api-access-9zbs2") pod "b93349aa-3f07-4290-b233-765f6b96a208" (UID: "b93349aa-3f07-4290-b233-765f6b96a208"). InnerVolumeSpecName "kube-api-access-9zbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:15:18.830829 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:18.830781    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zbs2\" (UniqueName: \"kubernetes.io/projected/b93349aa-3f07-4290-b233-765f6b96a208-kube-api-access-9zbs2\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:15:19.444128 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:19.444087    2582 generic.go:356] "Generic (PLEG): container finished" podID="b93349aa-3f07-4290-b233-765f6b96a208" containerID="ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de" exitCode=137
Sep 18 19:15:19.444128 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:19.444131    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:19.444645 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:19.444191    2582 scope.go:117] "RemoveContainer" containerID="ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de"
Sep 18 19:15:19.453744 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:19.453697    2582 scope.go:117] "RemoveContainer" containerID="ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de"
Sep 18 19:15:19.454043 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:15:19.454023    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de\": container with ID starting with ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de not found: ID does not exist" containerID="ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de"
Sep 18 19:15:19.454097 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:19.454054    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de"} err="failed to get container status \"ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de\": rpc error: code = NotFound desc = could not find container \"ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de\": container with ID starting with ac1d68105e189cb38a84a2963df2fa32e5bc70a868d56691bac5168a250810de not found: ID does not exist"
Sep 18 19:15:19.457944 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:15:19.457916    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-b4d4e0\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="b93349aa-3f07-4290-b233-765f6b96a208" pod="opendatahub/test-curl-7684-b4d4e0"
Sep 18 19:15:19.674762 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:19.674727    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:20.335849 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:20.335813    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93349aa-3f07-4290-b233-765f6b96a208" path="/var/lib/kubelet/pods/b93349aa-3f07-4290-b233-765f6b96a208/volumes"
Sep 18 19:15:20.636789 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:20.636759    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:20.687826 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:20.687790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:20.749910 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:20.749833    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4x89\" (UniqueName: \"kubernetes.io/projected/ab9c4404-7497-4142-a127-9d874f29b6bb-kube-api-access-h4x89\") pod \"ab9c4404-7497-4142-a127-9d874f29b6bb\" (UID: \"ab9c4404-7497-4142-a127-9d874f29b6bb\") "
Sep 18 19:15:20.752293 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:20.752259    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9c4404-7497-4142-a127-9d874f29b6bb-kube-api-access-h4x89" (OuterVolumeSpecName: "kube-api-access-h4x89") pod "ab9c4404-7497-4142-a127-9d874f29b6bb" (UID: "ab9c4404-7497-4142-a127-9d874f29b6bb"). InnerVolumeSpecName "kube-api-access-h4x89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:15:20.851120 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:20.851088    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4x89\" (UniqueName: \"kubernetes.io/projected/ab9c4404-7497-4142-a127-9d874f29b6bb-kube-api-access-h4x89\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:15:21.457618 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:21.457577    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab9c4404-7497-4142-a127-9d874f29b6bb" containerID="6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf" exitCode=137
Sep 18 19:15:21.457855 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:21.457637    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:21.457855 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:21.457687    2582 scope.go:117] "RemoveContainer" containerID="6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf"
Sep 18 19:15:21.467136 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:21.467114    2582 scope.go:117] "RemoveContainer" containerID="6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf"
Sep 18 19:15:21.467443 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:15:21.467422    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf\": container with ID starting with 6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf not found: ID does not exist" containerID="6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf"
Sep 18 19:15:21.467536 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:21.467451    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf"} err="failed to get container status \"6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf\": rpc error: code = NotFound desc = could not find container \"6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf\": container with ID starting with 6533efcaa53b460f3bca326d1c3f051cdb66834b8145eeb0f4dae66d34b464bf not found: ID does not exist"
Sep 18 19:15:21.474241 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:15:21.474213    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-51e4d0\" is forbidden: User \"system:node:ip-10-0-132-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-168.ec2.internal' and this object" podUID="ab9c4404-7497-4142-a127-9d874f29b6bb" pod="opendatahub/test-curl-7684-51e4d0"
Sep 18 19:15:22.335775 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:22.335739    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab9c4404-7497-4142-a127-9d874f29b6bb" path="/var/lib/kubelet/pods/ab9c4404-7497-4142-a127-9d874f29b6bb/volumes"
Sep 18 19:15:23.552109 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:23.552067    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:15:27.295031 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.294991    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"]
Sep 18 19:15:27.295471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.295379    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9c4404-7497-4142-a127-9d874f29b6bb" containerName="test-curl-7684-51e4d0"
Sep 18 19:15:27.295471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.295390    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9c4404-7497-4142-a127-9d874f29b6bb" containerName="test-curl-7684-51e4d0"
Sep 18 19:15:27.295471 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.295462    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab9c4404-7497-4142-a127-9d874f29b6bb" containerName="test-curl-7684-51e4d0"
Sep 18 19:15:27.299714 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.299694    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.305174 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.305144    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"default-dockercfg-5f268\""
Sep 18 19:15:27.305393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.305143    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"rate-test-model-feb80d-kserve-self-signed-certs\""
Sep 18 19:15:27.305393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.305144    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"openshift-service-ca.crt\""
Sep 18 19:15:27.307113 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.307092    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-feb80d\"/\"kube-root-ca.crt\""
Sep 18 19:15:27.323248 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.323209    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"]
Sep 18 19:15:27.379307 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.379263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:27.417913 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.417850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tmp-dir\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.418102 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.417924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tls-certs\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.418102 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.417979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8mll\" (UniqueName: \"kubernetes.io/projected/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-kube-api-access-p8mll\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.418102 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.418032    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-home\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.418214 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.418100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-dshm\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.418214 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.418147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-model-cache\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.518970 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.518925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tmp-dir\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519175 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tls-certs\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519175 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mll\" (UniqueName: \"kubernetes.io/projected/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-kube-api-access-p8mll\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519175 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-home\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519175 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-dshm\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519394 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-model-cache\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519487 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519454    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tmp-dir\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519551 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-home\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.519551 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.519531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-model-cache\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.521666 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.521639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-dshm\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.521954 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.521936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tls-certs\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.541835 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.541805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mll\" (UniqueName: \"kubernetes.io/projected/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-kube-api-access-p8mll\") pod \"rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") " pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.609682 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.609565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:27.895544 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:27.895489    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"]
Sep 18 19:15:27.898158 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:15:27.898123    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dd1aa0a_f7b7_430b_931c_5ced351aa6be.slice/crio-7efa2cc6e24c6856afc0a2bf2d56427efbfd81d7998f9f953f066f79dc1a3a8d WatchSource:0}: Error finding container 7efa2cc6e24c6856afc0a2bf2d56427efbfd81d7998f9f953f066f79dc1a3a8d: Status 404 returned error can't find the container with id 7efa2cc6e24c6856afc0a2bf2d56427efbfd81d7998f9f953f066f79dc1a3a8d
Sep 18 19:15:28.495704 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:28.495658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5" event={"ID":"1dd1aa0a-f7b7-430b-931c-5ced351aa6be","Type":"ContainerStarted","Data":"e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d"}
Sep 18 19:15:28.495704 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:28.495712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5" event={"ID":"1dd1aa0a-f7b7-430b-931c-5ced351aa6be","Type":"ContainerStarted","Data":"7efa2cc6e24c6856afc0a2bf2d56427efbfd81d7998f9f953f066f79dc1a3a8d"}
Sep 18 19:15:28.532761 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:28.532703    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5" podStartSLOduration=1.532685616 podStartE2EDuration="1.532685616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:28.530581973 +0000 UTC m=+1852.323258909" watchObservedRunningTime="2026-09-18 19:15:28.532685616 +0000 UTC m=+1852.325362611"
Sep 18 19:15:32.319755 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:32.319714    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:15:37.609741 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:37.609707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:37.610209 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:37.609754    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:37.624672 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:37.624639    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:38.559423 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:38.559382    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:15:44.694868 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:44.694822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:47.526335 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:47.526293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:15:48.569459 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:15:48.569415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:16:14.010557 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:14.010515    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:16:24.214489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:24.214442    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:16:30.168768 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:30.168713    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"]
Sep 18 19:16:30.215670 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:30.215630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:31.307643 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:31.307597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:31.812211 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:31.812170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:32.183295 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:32.183263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:32.486290 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:32.486199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:35.140138 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:35.140099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:35.147762 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:35.147730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:16:35.767469 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:35.767431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:36.188458 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:36.188426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:37.198649 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:37.198581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:38.621913 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:38.621865    2582 scope.go:117] "RemoveContainer" containerID="003f0105a71d772d6d25c2f6ece1c4dc991f3cd144a486fa366f43ed45ef62ba"
Sep 18 19:16:39.101820 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:39.101784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:40.117405 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.117369    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"]
Sep 18 19:16:40.122181 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.122159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.125976 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.125952    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f8cdfc-serving-cert\""
Sep 18 19:16:40.144770 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.144732    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"]
Sep 18 19:16:40.186355 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.186321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:40.216449 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.216415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-metrics-certs\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.216618 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.216465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda315c3-0224-482c-b69f-e575e80865ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.216618 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.216542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9g2\" (UniqueName: \"kubernetes.io/projected/bda315c3-0224-482c-b69f-e575e80865ad-kube-api-access-ml9g2\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.216618 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.216562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.294344 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.294313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:40.317540 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.317507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-metrics-certs\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.317625 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.317562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda315c3-0224-482c-b69f-e575e80865ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.317686 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.317638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9g2\" (UniqueName: \"kubernetes.io/projected/bda315c3-0224-482c-b69f-e575e80865ad-kube-api-access-ml9g2\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.317686 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.317660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.318367 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.318267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda315c3-0224-482c-b69f-e575e80865ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.320254 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.320233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.320343 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.320272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-metrics-certs\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.330746 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.330725    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9g2\" (UniqueName: \"kubernetes.io/projected/bda315c3-0224-482c-b69f-e575e80865ad-kube-api-access-ml9g2\") pod \"maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.432490 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.432457    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:40.731945 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.731861    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"]
Sep 18 19:16:40.734287 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:16:40.734244    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda315c3_0224_482c_b69f_e575e80865ad.slice/crio-722cbc2e0d1987056df7b5b855761f3c85059b8797b76cb13a44346ba35a3622 WatchSource:0}: Error finding container 722cbc2e0d1987056df7b5b855761f3c85059b8797b76cb13a44346ba35a3622: Status 404 returned error can't find the container with id 722cbc2e0d1987056df7b5b855761f3c85059b8797b76cb13a44346ba35a3622
Sep 18 19:16:40.735645 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.735627    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:16:40.876396 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.876345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh" event={"ID":"bda315c3-0224-482c-b69f-e575e80865ad","Type":"ContainerStarted","Data":"722cbc2e0d1987056df7b5b855761f3c85059b8797b76cb13a44346ba35a3622"}
Sep 18 19:16:40.959629 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:40.959577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:41.191693 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.191667    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:16:41.358349 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.358312    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"]
Sep 18 19:16:41.362057 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.362034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.368905 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.368862    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f8cdfc\""
Sep 18 19:16:41.369059 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.368918    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f8cdfc-dockercfg-j6z9p\""
Sep 18 19:16:41.379122 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.379099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"]
Sep 18 19:16:41.429306 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.429274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrn2d\" (UniqueName: \"kubernetes.io/projected/1cd53805-7c34-4e51-b52c-5961c97868c0-kube-api-access-xrn2d\") pod \"payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.429474 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.429319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd53805-7c34-4e51-b52c-5961c97868c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.530478 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.530443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrn2d\" (UniqueName: \"kubernetes.io/projected/1cd53805-7c34-4e51-b52c-5961c97868c0-kube-api-access-xrn2d\") pod \"payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.530478 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.530487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd53805-7c34-4e51-b52c-5961c97868c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.531262 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.531241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd53805-7c34-4e51-b52c-5961c97868c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.540513 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.540483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrn2d\" (UniqueName: \"kubernetes.io/projected/1cd53805-7c34-4e51-b52c-5961c97868c0-kube-api-access-xrn2d\") pod \"payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.674515 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.674477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:41.883836 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.883801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh" event={"ID":"bda315c3-0224-482c-b69f-e575e80865ad","Type":"ContainerStarted","Data":"18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5"}
Sep 18 19:16:41.884031 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.883866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:41.954686 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:41.954569    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh" podStartSLOduration=1.557710648 podStartE2EDuration="1.954551308s" totalImagesPullingTime="396.84066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:41.952305774 +0000 UTC m=+1925.744982717" watchObservedRunningTime="2026-09-18 19:16:41.954551308 +0000 UTC m=+1925.747228238"
Sep 18 19:16:42.014443 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:16:42.014406    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd53805_7c34_4e51_b52c_5961c97868c0.slice/crio-1ba8f1c4fd35a013acece1c34c5fe0fc360ae3600a564793a724856c88564d50 WatchSource:0}: Error finding container 1ba8f1c4fd35a013acece1c34c5fe0fc360ae3600a564793a724856c88564d50: Status 404 returned error can't find the container with id 1ba8f1c4fd35a013acece1c34c5fe0fc360ae3600a564793a724856c88564d50
Sep 18 19:16:42.018914 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:42.018866    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"]
Sep 18 19:16:42.890424 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:42.890365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll" event={"ID":"1cd53805-7c34-4e51-b52c-5961c97868c0","Type":"ContainerStarted","Data":"4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2"}
Sep 18 19:16:42.890977 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:42.890433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll" event={"ID":"1cd53805-7c34-4e51-b52c-5961c97868c0","Type":"ContainerStarted","Data":"1ba8f1c4fd35a013acece1c34c5fe0fc360ae3600a564793a724856c88564d50"}
Sep 18 19:16:42.890977 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:42.890645    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:16:42.911985 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:42.911932    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll" podStartSLOduration=1.9119172770000001 podStartE2EDuration="1.911917277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:42.908905879 +0000 UTC m=+1926.701582816" watchObservedRunningTime="2026-09-18 19:16:42.911917277 +0000 UTC m=+1926.704594264"
Sep 18 19:16:45.188655 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.188597    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5" podUID="1dd1aa0a-f7b7-430b-931c-5ced351aa6be" containerName="main" containerID="cri-o://e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d" gracePeriod=45
Sep 18 19:16:45.246906 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.246844    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:16:45.463998 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.463967    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:16:45.570575 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570537    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-home\") pod \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") "
Sep 18 19:16:45.570575 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570583    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tmp-dir\") pod \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") "
Sep 18 19:16:45.570906 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570609    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-model-cache\") pod \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") "
Sep 18 19:16:45.570906 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tls-certs\") pod \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") "
Sep 18 19:16:45.570906 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570705    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8mll\" (UniqueName: \"kubernetes.io/projected/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-kube-api-access-p8mll\") pod \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") "
Sep 18 19:16:45.570906 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570757    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-dshm\") pod \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\" (UID: \"1dd1aa0a-f7b7-430b-931c-5ced351aa6be\") "
Sep 18 19:16:45.571119 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570922    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dd1aa0a-f7b7-430b-931c-5ced351aa6be" (UID: "1dd1aa0a-f7b7-430b-931c-5ced351aa6be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:45.571119 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.570939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dd1aa0a-f7b7-430b-931c-5ced351aa6be" (UID: "1dd1aa0a-f7b7-430b-931c-5ced351aa6be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:45.571221 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.571150    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:16:45.571221 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.571174    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:16:45.571221 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.571195    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-home" (OuterVolumeSpecName: "home") pod "1dd1aa0a-f7b7-430b-931c-5ced351aa6be" (UID: "1dd1aa0a-f7b7-430b-931c-5ced351aa6be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:45.573265 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.573216    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dd1aa0a-f7b7-430b-931c-5ced351aa6be" (UID: "1dd1aa0a-f7b7-430b-931c-5ced351aa6be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:16:45.573473 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.573456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-kube-api-access-p8mll" (OuterVolumeSpecName: "kube-api-access-p8mll") pod "1dd1aa0a-f7b7-430b-931c-5ced351aa6be" (UID: "1dd1aa0a-f7b7-430b-931c-5ced351aa6be"). InnerVolumeSpecName "kube-api-access-p8mll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:16:45.573543 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.573515    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-dshm" (OuterVolumeSpecName: "dshm") pod "1dd1aa0a-f7b7-430b-931c-5ced351aa6be" (UID: "1dd1aa0a-f7b7-430b-931c-5ced351aa6be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:16:45.672766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.672727    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:16:45.672766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.672761    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:16:45.672766 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.672773    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:16:45.673085 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.672790    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8mll\" (UniqueName: \"kubernetes.io/projected/1dd1aa0a-f7b7-430b-931c-5ced351aa6be-kube-api-access-p8mll\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:16:45.913536 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.913493    2582 generic.go:356] "Generic (PLEG): container finished" podID="1dd1aa0a-f7b7-430b-931c-5ced351aa6be" containerID="e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d" exitCode=0
Sep 18 19:16:45.913754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.913540    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"
Sep 18 19:16:45.913754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.913586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5" event={"ID":"1dd1aa0a-f7b7-430b-931c-5ced351aa6be","Type":"ContainerDied","Data":"e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d"}
Sep 18 19:16:45.913754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.913646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5" event={"ID":"1dd1aa0a-f7b7-430b-931c-5ced351aa6be","Type":"ContainerDied","Data":"7efa2cc6e24c6856afc0a2bf2d56427efbfd81d7998f9f953f066f79dc1a3a8d"}
Sep 18 19:16:45.913754 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.913670    2582 scope.go:117] "RemoveContainer" containerID="e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d"
Sep 18 19:16:45.923689 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.923672    2582 scope.go:117] "RemoveContainer" containerID="e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d"
Sep 18 19:16:45.923989 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:16:45.923963    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d\": container with ID starting with e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d not found: ID does not exist" containerID="e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d"
Sep 18 19:16:45.924061 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.924001    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d"} err="failed to get container status \"e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d\": rpc error: code = NotFound desc = could not find container \"e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d\": container with ID starting with e3ef9046f5ac89ce10b9f5b665fc469d5d410950db3cd15867131b9bd19cec7d not found: ID does not exist"
Sep 18 19:16:45.938833 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.938797    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"]
Sep 18 19:16:45.940863 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:45.940840    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-feb80d/rate-test-model-feb80d-kserve-7c6bc789cf-8zcm5"]
Sep 18 19:16:46.335442 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:46.335340    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd1aa0a-f7b7-430b-931c-5ced351aa6be" path="/var/lib/kubelet/pods/1dd1aa0a-f7b7-430b-931c-5ced351aa6be/volumes"
Sep 18 19:16:47.896083 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:47.896053    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:16:53.905074 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:16:53.905039    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:17:05.875365 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:05.875330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:06.014363 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:06.014325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:06.654511 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:06.654480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:07.270040 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:07.269999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:07.837331 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:07.837295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:08.076044 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:08.076006    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:09.095641 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:09.095605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:10.335931 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:10.335902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:11.608956 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:11.608923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:11.930182 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:11.930140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:12.257669 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:12.257578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:13.726218 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:13.726171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:15.110306 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:15.110262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:16.429532 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:16.429482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:16.637561 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:16.637509    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:18.047872 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:18.047799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:18.171780 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:18.171732    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:18.341601 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:18.341507    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:19.090206 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.090120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:19.233333 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.233291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"]
Sep 18 19:17:19.233707 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.233695    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd1aa0a-f7b7-430b-931c-5ced351aa6be" containerName="main"
Sep 18 19:17:19.233752 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.233709    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd1aa0a-f7b7-430b-931c-5ced351aa6be" containerName="main"
Sep 18 19:17:19.233796 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.233782    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd1aa0a-f7b7-430b-931c-5ced351aa6be" containerName="main"
Sep 18 19:17:19.237215 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.237195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.240232 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.240206    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8f9642-dockercfg-pw244\""
Sep 18 19:17:19.240606 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.240584    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8f9642\""
Sep 18 19:17:19.278869 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.278836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"]
Sep 18 19:17:19.297694 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.297659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzhv\" (UniqueName: \"kubernetes.io/projected/8c482324-3b11-4231-882f-c53c876eec57-kube-api-access-bwzhv\") pod \"payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.297914 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.297714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c482324-3b11-4231-882f-c53c876eec57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.373672 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.373567    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"]
Sep 18 19:17:19.377266 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.377239    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.391285 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.391257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:19.399000 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.398970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c482324-3b11-4231-882f-c53c876eec57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.399151 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.399082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzhv\" (UniqueName: \"kubernetes.io/projected/8c482324-3b11-4231-882f-c53c876eec57-kube-api-access-bwzhv\") pod \"payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.399685 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.399665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c482324-3b11-4231-882f-c53c876eec57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.403954 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.403925    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"]
Sep 18 19:17:19.431447 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.431405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzhv\" (UniqueName: \"kubernetes.io/projected/8c482324-3b11-4231-882f-c53c876eec57-kube-api-access-bwzhv\") pod \"payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.500288 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.500252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e29316-03b2-4824-bd7f-db9ac0b217d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.500288 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.500294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hff99\" (UniqueName: \"kubernetes.io/projected/e2e29316-03b2-4824-bd7f-db9ac0b217d2-kube-api-access-hff99\") pod \"payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.548217 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.548177    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:19.602553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.601911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e29316-03b2-4824-bd7f-db9ac0b217d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.602553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.601976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hff99\" (UniqueName: \"kubernetes.io/projected/e2e29316-03b2-4824-bd7f-db9ac0b217d2-kube-api-access-hff99\") pod \"payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.602943 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.602922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e29316-03b2-4824-bd7f-db9ac0b217d2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.617019 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.616947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hff99\" (UniqueName: \"kubernetes.io/projected/e2e29316-03b2-4824-bd7f-db9ac0b217d2-kube-api-access-hff99\") pod \"payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.688060 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.688016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:19.849148 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.849096    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"]
Sep 18 19:17:19.851472 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:17:19.851444    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c482324_3b11_4231_882f_c53c876eec57.slice/crio-6b6e6675615e47e1513d86f5cd21947dd85f5feee0acb6c9d118a736b76c4d6f WatchSource:0}: Error finding container 6b6e6675615e47e1513d86f5cd21947dd85f5feee0acb6c9d118a736b76c4d6f: Status 404 returned error can't find the container with id 6b6e6675615e47e1513d86f5cd21947dd85f5feee0acb6c9d118a736b76c4d6f
Sep 18 19:17:19.988998 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:19.988959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"]
Sep 18 19:17:19.991614 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:17:19.991585    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e29316_03b2_4824_bd7f_db9ac0b217d2.slice/crio-71310e818898e2025d154f798240a65b8d22964788d1c79fa3de645ba17179b2 WatchSource:0}: Error finding container 71310e818898e2025d154f798240a65b8d22964788d1c79fa3de645ba17179b2: Status 404 returned error can't find the container with id 71310e818898e2025d154f798240a65b8d22964788d1c79fa3de645ba17179b2
Sep 18 19:17:20.095275 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.095238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr" event={"ID":"e2e29316-03b2-4824-bd7f-db9ac0b217d2","Type":"ContainerStarted","Data":"ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295"}
Sep 18 19:17:20.095905 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.095304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr" event={"ID":"e2e29316-03b2-4824-bd7f-db9ac0b217d2","Type":"ContainerStarted","Data":"71310e818898e2025d154f798240a65b8d22964788d1c79fa3de645ba17179b2"}
Sep 18 19:17:20.095905 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.095427    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:20.096992 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.096968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv" event={"ID":"8c482324-3b11-4231-882f-c53c876eec57","Type":"ContainerStarted","Data":"6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f"}
Sep 18 19:17:20.097124 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.097005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv" event={"ID":"8c482324-3b11-4231-882f-c53c876eec57","Type":"ContainerStarted","Data":"6b6e6675615e47e1513d86f5cd21947dd85f5feee0acb6c9d118a736b76c4d6f"}
Sep 18 19:17:20.097124 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.097090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:20.119093 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.119028    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr" podStartSLOduration=1.119010031 podStartE2EDuration="1.119010031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:20.11800743 +0000 UTC m=+1963.910684398" watchObservedRunningTime="2026-09-18 19:17:20.119010031 +0000 UTC m=+1963.911686962"
Sep 18 19:17:20.155094 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:20.155035    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv" podStartSLOduration=1.15501885 podStartE2EDuration="1.15501885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:20.152138914 +0000 UTC m=+1963.944815902" watchObservedRunningTime="2026-09-18 19:17:20.15501885 +0000 UTC m=+1963.947695793"
Sep 18 19:17:23.644177 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:23.644140    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:17:31.104570 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:31.104529    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:17:31.105089 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:31.104620    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:17:31.802581 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:31.802544    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:41.964820 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:41.964770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:42.896108 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:42.896060    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:43.804904 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:43.804858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:43.980135 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:43.980091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:44.811521 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:44.811484    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:46.228847 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:46.228813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:47.595273 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:47.595231    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr_e2e29316-03b2-4824-bd7f-db9ac0b217d2/payload-processing/0.log"
Sep 18 19:17:47.903386 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:47.903344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:49.589332 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:49.589294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:53.772394 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:53.772357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:17:57.978313 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:17:57.978264    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:18:09.851212 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:09.851174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:22.830705 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:22.830663    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:18:24.494128 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:24.494090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:29.490711 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:29.490681    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr_e2e29316-03b2-4824-bd7f-db9ac0b217d2/payload-processing/0.log"
Sep 18 19:18:29.637380 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:29.637352    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr_e2e29316-03b2-4824-bd7f-db9ac0b217d2/payload-processing/0.log"
Sep 18 19:18:30.055977 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:30.055943    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:30.462542 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:30.462508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:30.953205 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:30.953158    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:32.885074 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:32.885042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:33.507559 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:33.507523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:34.153866 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:34.153822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:34.536941 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:34.536810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:36.363284 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:36.363248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:36.607660 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:36.607626    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:37.844676 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:37.844644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:39.376278 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:39.376244    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:40.625647 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:40.625613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:40.818158 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:40.818125    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:41.034485 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.034440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:41.578413 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.578367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:41.747315 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.747271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:18:41.838944 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.838847    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"]
Sep 18 19:18:41.842343 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.842313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:41.845845 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.845821    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-790775-dockercfg-cq6wz\""
Sep 18 19:18:41.846002 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.845821    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-790775\""
Sep 18 19:18:41.906535 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.906501    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"]
Sep 18 19:18:41.965412 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.965360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrp9\" (UniqueName: \"kubernetes.io/projected/65379eeb-7ccc-4990-bf7c-f1c455c518d8-kube-api-access-mcrp9\") pod \"payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:41.965596 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:41.965436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65379eeb-7ccc-4990-bf7c-f1c455c518d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:42.066498 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.066464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65379eeb-7ccc-4990-bf7c-f1c455c518d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:42.066686 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.066598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrp9\" (UniqueName: \"kubernetes.io/projected/65379eeb-7ccc-4990-bf7c-f1c455c518d8-kube-api-access-mcrp9\") pod \"payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:42.067245 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.067226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65379eeb-7ccc-4990-bf7c-f1c455c518d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:42.079851 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.079828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrp9\" (UniqueName: \"kubernetes.io/projected/65379eeb-7ccc-4990-bf7c-f1c455c518d8-kube-api-access-mcrp9\") pod \"payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:42.153213 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.153178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:42.445716 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.445651    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"]
Sep 18 19:18:42.448096 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:18:42.448066    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65379eeb_7ccc_4990_bf7c_f1c455c518d8.slice/crio-cca74ebcdcf3082d223070c5ed1089a3f54918415ee85d00bdd3e8a9b565a45b WatchSource:0}: Error finding container cca74ebcdcf3082d223070c5ed1089a3f54918415ee85d00bdd3e8a9b565a45b: Status 404 returned error can't find the container with id cca74ebcdcf3082d223070c5ed1089a3f54918415ee85d00bdd3e8a9b565a45b
Sep 18 19:18:42.531670 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.531623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x" event={"ID":"65379eeb-7ccc-4990-bf7c-f1c455c518d8","Type":"ContainerStarted","Data":"ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a"}
Sep 18 19:18:42.531810 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.531710    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:18:42.531810 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.531740    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x" event={"ID":"65379eeb-7ccc-4990-bf7c-f1c455c518d8","Type":"ContainerStarted","Data":"cca74ebcdcf3082d223070c5ed1089a3f54918415ee85d00bdd3e8a9b565a45b"}
Sep 18 19:18:42.600692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:42.600621    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x" podStartSLOduration=1.600596371 podStartE2EDuration="1.600596371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:42.597413871 +0000 UTC m=+2046.390090822" watchObservedRunningTime="2026-09-18 19:18:42.600596371 +0000 UTC m=+2046.393273305"
Sep 18 19:18:45.979987 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:45.979947    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:18:53.538764 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:18:53.538681    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:19:05.470717 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:05.470677    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:06.954192 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:06.954143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:08.133716 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:08.133672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:08.732545 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:08.732494    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:09.673800 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:09.673765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:11.354299 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:11.354264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:13.198071 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:13.198033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:14.483863 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:14.483821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:15.884292 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:15.884255    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:16.986442 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:16.986407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:18.442380 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:18.442349    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:19.736341 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:19.736306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:20.884597 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:20.884560    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:20.983219 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:20.983182    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"]
Sep 18 19:19:20.983456 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:20.983434    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x" podUID="65379eeb-7ccc-4990-bf7c-f1c455c518d8" containerName="payload-processing" containerID="cri-o://ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a" gracePeriod=30
Sep 18 19:19:21.128445 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.128404    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:21.230756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.230731    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:19:21.270564 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.270539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:21.332188 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.332161    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65379eeb-7ccc-4990-bf7c-f1c455c518d8-plugins-config-volume\") pod \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") "
Sep 18 19:19:21.332335 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.332255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcrp9\" (UniqueName: \"kubernetes.io/projected/65379eeb-7ccc-4990-bf7c-f1c455c518d8-kube-api-access-mcrp9\") pod \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\" (UID: \"65379eeb-7ccc-4990-bf7c-f1c455c518d8\") "
Sep 18 19:19:21.332591 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.332563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65379eeb-7ccc-4990-bf7c-f1c455c518d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65379eeb-7ccc-4990-bf7c-f1c455c518d8" (UID: "65379eeb-7ccc-4990-bf7c-f1c455c518d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:21.332718 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.332697    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65379eeb-7ccc-4990-bf7c-f1c455c518d8-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:21.334835 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.334799    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65379eeb-7ccc-4990-bf7c-f1c455c518d8-kube-api-access-mcrp9" (OuterVolumeSpecName: "kube-api-access-mcrp9") pod "65379eeb-7ccc-4990-bf7c-f1c455c518d8" (UID: "65379eeb-7ccc-4990-bf7c-f1c455c518d8"). InnerVolumeSpecName "kube-api-access-mcrp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:19:21.433989 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.433910    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcrp9\" (UniqueName: \"kubernetes.io/projected/65379eeb-7ccc-4990-bf7c-f1c455c518d8-kube-api-access-mcrp9\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:21.501302 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.501271    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:21.735605 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.735517    2582 generic.go:356] "Generic (PLEG): container finished" podID="65379eeb-7ccc-4990-bf7c-f1c455c518d8" containerID="ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a" exitCode=0
Sep 18 19:19:21.735605 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.735556    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"
Sep 18 19:19:21.735605 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.735581    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x" event={"ID":"65379eeb-7ccc-4990-bf7c-f1c455c518d8","Type":"ContainerDied","Data":"ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a"}
Sep 18 19:19:21.735919 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.735622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x" event={"ID":"65379eeb-7ccc-4990-bf7c-f1c455c518d8","Type":"ContainerDied","Data":"cca74ebcdcf3082d223070c5ed1089a3f54918415ee85d00bdd3e8a9b565a45b"}
Sep 18 19:19:21.735919 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.735643    2582 scope.go:117] "RemoveContainer" containerID="ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a"
Sep 18 19:19:21.744520 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.744499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:21.745171 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.745156    2582 scope.go:117] "RemoveContainer" containerID="ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a"
Sep 18 19:19:21.745444 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:19:21.745423    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a\": container with ID starting with ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a not found: ID does not exist" containerID="ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a"
Sep 18 19:19:21.745505 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.745454    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a"} err="failed to get container status \"ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a\": rpc error: code = NotFound desc = could not find container \"ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a\": container with ID starting with ec5bff7c2be739505596e1e48ca965c975256f9841b791722e54207e958ffd3a not found: ID does not exist"
Sep 18 19:19:21.775408 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.775386    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"]
Sep 18 19:19:21.791949 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:21.791918    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-790775-769fcbbfcc-2h59x"]
Sep 18 19:19:22.051059 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:22.050977    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:22.272087 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:22.272053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:22.335212 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:22.335148    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65379eeb-7ccc-4990-bf7c-f1c455c518d8" path="/var/lib/kubelet/pods/65379eeb-7ccc-4990-bf7c-f1c455c518d8/volumes"
Sep 18 19:19:23.755176 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:23.755136    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:25.170135 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:25.170106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:25.843530 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:25.843499    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:19:27.526598 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:27.526562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:29.263842 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:29.263810    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:31.808756 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:31.808719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:31.952433 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:31.952394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:32.217829 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:32.217791    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:36.431988 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:36.431952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:37.839657 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:37.839618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:38.135790 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:38.135703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:39.279075 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:39.279034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:40.865684 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:40.865645    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:42.041653 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:42.041617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:43.601759 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:43.601719    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:50.032708 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:50.032672    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:19:52.138692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.138655    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"]
Sep 18 19:19:52.139174 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.138858    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh" podUID="bda315c3-0224-482c-b69f-e575e80865ad" containerName="maas-api" containerID="cri-o://18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5" gracePeriod=30
Sep 18 19:19:52.224169 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.224133    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:52.309318 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.309286    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"]
Sep 18 19:19:52.309523 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.309486    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll" podUID="1cd53805-7c34-4e51-b52c-5961c97868c0" containerName="payload-pre-processing" containerID="cri-o://4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2" gracePeriod=30
Sep 18 19:19:52.442622 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.442598    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:19:52.507044 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.507012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:52.549453 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.549430    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:19:52.620715 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.620685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml9g2\" (UniqueName: \"kubernetes.io/projected/bda315c3-0224-482c-b69f-e575e80865ad-kube-api-access-ml9g2\") pod \"bda315c3-0224-482c-b69f-e575e80865ad\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") "
Sep 18 19:19:52.620923 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.620723    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-metrics-certs\") pod \"bda315c3-0224-482c-b69f-e575e80865ad\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") "
Sep 18 19:19:52.620923 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.620758    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda315c3-0224-482c-b69f-e575e80865ad-openshift-service-ca\") pod \"bda315c3-0224-482c-b69f-e575e80865ad\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") "
Sep 18 19:19:52.620923 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.620773    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-maas-api-tls\") pod \"bda315c3-0224-482c-b69f-e575e80865ad\" (UID: \"bda315c3-0224-482c-b69f-e575e80865ad\") "
Sep 18 19:19:52.621195 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.621163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda315c3-0224-482c-b69f-e575e80865ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bda315c3-0224-482c-b69f-e575e80865ad" (UID: "bda315c3-0224-482c-b69f-e575e80865ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:52.623103 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.623068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bda315c3-0224-482c-b69f-e575e80865ad" (UID: "bda315c3-0224-482c-b69f-e575e80865ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:52.623222 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.623105    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bda315c3-0224-482c-b69f-e575e80865ad" (UID: "bda315c3-0224-482c-b69f-e575e80865ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:19:52.623222 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.623161    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda315c3-0224-482c-b69f-e575e80865ad-kube-api-access-ml9g2" (OuterVolumeSpecName: "kube-api-access-ml9g2") pod "bda315c3-0224-482c-b69f-e575e80865ad" (UID: "bda315c3-0224-482c-b69f-e575e80865ad"). InnerVolumeSpecName "kube-api-access-ml9g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:19:52.713669 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.713588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:52.721734 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.721711    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrn2d\" (UniqueName: \"kubernetes.io/projected/1cd53805-7c34-4e51-b52c-5961c97868c0-kube-api-access-xrn2d\") pod \"1cd53805-7c34-4e51-b52c-5961c97868c0\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") "
Sep 18 19:19:52.721825 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.721760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd53805-7c34-4e51-b52c-5961c97868c0-plugins-config-volume\") pod \"1cd53805-7c34-4e51-b52c-5961c97868c0\" (UID: \"1cd53805-7c34-4e51-b52c-5961c97868c0\") "
Sep 18 19:19:52.721961 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.721950    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-metrics-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:52.722003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.721967    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda315c3-0224-482c-b69f-e575e80865ad-openshift-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:52.722003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.721981    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bda315c3-0224-482c-b69f-e575e80865ad-maas-api-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:52.722003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.721991    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml9g2\" (UniqueName: \"kubernetes.io/projected/bda315c3-0224-482c-b69f-e575e80865ad-kube-api-access-ml9g2\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:52.722209 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.722185    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd53805-7c34-4e51-b52c-5961c97868c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cd53805-7c34-4e51-b52c-5961c97868c0" (UID: "1cd53805-7c34-4e51-b52c-5961c97868c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:19:52.724128 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.724107    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd53805-7c34-4e51-b52c-5961c97868c0-kube-api-access-xrn2d" (OuterVolumeSpecName: "kube-api-access-xrn2d") pod "1cd53805-7c34-4e51-b52c-5961c97868c0" (UID: "1cd53805-7c34-4e51-b52c-5961c97868c0"). InnerVolumeSpecName "kube-api-access-xrn2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:19:52.822728 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.822694    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cd53805-7c34-4e51-b52c-5961c97868c0-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:52.822728 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.822722    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrn2d\" (UniqueName: \"kubernetes.io/projected/1cd53805-7c34-4e51-b52c-5961c97868c0-kube-api-access-xrn2d\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:19:52.878960 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.878930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:52.893438 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.893410    2582 generic.go:356] "Generic (PLEG): container finished" podID="1cd53805-7c34-4e51-b52c-5961c97868c0" containerID="4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2" exitCode=0
Sep 18 19:19:52.893574 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.893462    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"
Sep 18 19:19:52.893574 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.893500    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll" event={"ID":"1cd53805-7c34-4e51-b52c-5961c97868c0","Type":"ContainerDied","Data":"4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2"}
Sep 18 19:19:52.893680 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.893587    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll" event={"ID":"1cd53805-7c34-4e51-b52c-5961c97868c0","Type":"ContainerDied","Data":"1ba8f1c4fd35a013acece1c34c5fe0fc360ae3600a564793a724856c88564d50"}
Sep 18 19:19:52.893680 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.893621    2582 scope.go:117] "RemoveContainer" containerID="4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2"
Sep 18 19:19:52.894941 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.894918    2582 generic.go:356] "Generic (PLEG): container finished" podID="bda315c3-0224-482c-b69f-e575e80865ad" containerID="18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5" exitCode=0
Sep 18 19:19:52.895046 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.894971    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"
Sep 18 19:19:52.895046 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.895005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh" event={"ID":"bda315c3-0224-482c-b69f-e575e80865ad","Type":"ContainerDied","Data":"18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5"}
Sep 18 19:19:52.895145 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.895059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh" event={"ID":"bda315c3-0224-482c-b69f-e575e80865ad","Type":"ContainerDied","Data":"722cbc2e0d1987056df7b5b855761f3c85059b8797b76cb13a44346ba35a3622"}
Sep 18 19:19:52.905955 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.905934    2582 scope.go:117] "RemoveContainer" containerID="4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2"
Sep 18 19:19:52.906275 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:19:52.906254    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2\": container with ID starting with 4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2 not found: ID does not exist" containerID="4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2"
Sep 18 19:19:52.906377 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.906285    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2"} err="failed to get container status \"4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2\": rpc error: code = NotFound desc = could not find container \"4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2\": container with ID starting with 4f403a5e63c18abcb082af7ee9928fe57a460f449a5ee622600c9a7747a4ccb2 not found: ID does not exist"
Sep 18 19:19:52.906377 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.906307    2582 scope.go:117] "RemoveContainer" containerID="18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5"
Sep 18 19:19:52.914771 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.914754    2582 scope.go:117] "RemoveContainer" containerID="18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5"
Sep 18 19:19:52.915065 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:19:52.915044    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5\": container with ID starting with 18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5 not found: ID does not exist" containerID="18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5"
Sep 18 19:19:52.915145 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.915076    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5"} err="failed to get container status \"18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5\": rpc error: code = NotFound desc = could not find container \"18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5\": container with ID starting with 18bb248a675b8f6c62b896bc6d057d5e8b2663e9d9d9231370d15aa04f01f3e5 not found: ID does not exist"
Sep 18 19:19:52.935930 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.935909    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"]
Sep 18 19:19:52.946060 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.946036    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f8cdfc-84bdd86fc4-pppll"]
Sep 18 19:19:52.979240 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.979181    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"]
Sep 18 19:19:52.985793 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:52.985774    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f8cdfc-dd4f56955-gw2zh"]
Sep 18 19:19:53.147552 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:53.147520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:53.665972 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:53.665940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:54.280777 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:54.280748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:54.334975 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:54.334943    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd53805-7c34-4e51-b52c-5961c97868c0" path="/var/lib/kubelet/pods/1cd53805-7c34-4e51-b52c-5961c97868c0/volumes"
Sep 18 19:19:54.335358 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:54.335345    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda315c3-0224-482c-b69f-e575e80865ad" path="/var/lib/kubelet/pods/bda315c3-0224-482c-b69f-e575e80865ad/volumes"
Sep 18 19:19:55.892591 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:55.892561    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:56.151480 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:56.151381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:57.999860 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:57.999825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:58.606191 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:58.606159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:19:59.416034 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:19:59.416000    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:04.739748 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:04.739714    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:20:34.675178 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:34.675096    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:37.159192 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:37.159154    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:45.312098 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:45.312064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:45.664389 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:45.664357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:45.886466 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:45.886431    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:46.623584 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:46.623550    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:49.637125 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:49.637090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:51.552898 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:51.552860    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:52.908176 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:52.908141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:54.633074 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:54.633032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:56.084049 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:56.084014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:58.254107 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.254080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:58.316275 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.316247    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"]
Sep 18 19:20:58.316450 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.316432    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr" podUID="e2e29316-03b2-4824-bd7f-db9ac0b217d2" containerName="payload-processing" containerID="cri-o://ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295" gracePeriod=30
Sep 18 19:20:58.324095 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.324070    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"]
Sep 18 19:20:58.324256 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.324238    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv" podUID="8c482324-3b11-4231-882f-c53c876eec57" containerName="payload-pre-processing" containerID="cri-o://6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f" gracePeriod=30
Sep 18 19:20:58.578740 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.578713    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:20:58.581657 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.581637    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:20:58.637799 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.637777    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:58.656110 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.656086    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hff99\" (UniqueName: \"kubernetes.io/projected/e2e29316-03b2-4824-bd7f-db9ac0b217d2-kube-api-access-hff99\") pod \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") "
Sep 18 19:20:58.656226 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.656145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c482324-3b11-4231-882f-c53c876eec57-plugins-config-volume\") pod \"8c482324-3b11-4231-882f-c53c876eec57\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") "
Sep 18 19:20:58.656287 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.656229    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e29316-03b2-4824-bd7f-db9ac0b217d2-plugins-config-volume\") pod \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\" (UID: \"e2e29316-03b2-4824-bd7f-db9ac0b217d2\") "
Sep 18 19:20:58.656287 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.656258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwzhv\" (UniqueName: \"kubernetes.io/projected/8c482324-3b11-4231-882f-c53c876eec57-kube-api-access-bwzhv\") pod \"8c482324-3b11-4231-882f-c53c876eec57\" (UID: \"8c482324-3b11-4231-882f-c53c876eec57\") "
Sep 18 19:20:58.656604 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.656578    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c482324-3b11-4231-882f-c53c876eec57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c482324-3b11-4231-882f-c53c876eec57" (UID: "8c482324-3b11-4231-882f-c53c876eec57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:20:58.656692 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.656637    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e29316-03b2-4824-bd7f-db9ac0b217d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2e29316-03b2-4824-bd7f-db9ac0b217d2" (UID: "e2e29316-03b2-4824-bd7f-db9ac0b217d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:20:58.658399 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.658380    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e29316-03b2-4824-bd7f-db9ac0b217d2-kube-api-access-hff99" (OuterVolumeSpecName: "kube-api-access-hff99") pod "e2e29316-03b2-4824-bd7f-db9ac0b217d2" (UID: "e2e29316-03b2-4824-bd7f-db9ac0b217d2"). InnerVolumeSpecName "kube-api-access-hff99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:20:58.658518 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.658498    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c482324-3b11-4231-882f-c53c876eec57-kube-api-access-bwzhv" (OuterVolumeSpecName: "kube-api-access-bwzhv") pod "8c482324-3b11-4231-882f-c53c876eec57" (UID: "8c482324-3b11-4231-882f-c53c876eec57"). InnerVolumeSpecName "kube-api-access-bwzhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:20:58.757307 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.757286    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwzhv\" (UniqueName: \"kubernetes.io/projected/8c482324-3b11-4231-882f-c53c876eec57-kube-api-access-bwzhv\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:20:58.757307 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.757306    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hff99\" (UniqueName: \"kubernetes.io/projected/e2e29316-03b2-4824-bd7f-db9ac0b217d2-kube-api-access-hff99\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:20:58.757442 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.757316    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c482324-3b11-4231-882f-c53c876eec57-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:20:58.757442 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:58.757331    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e29316-03b2-4824-bd7f-db9ac0b217d2-plugins-config-volume\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:20:59.002605 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.002577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:59.228688 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.228661    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2e29316-03b2-4824-bd7f-db9ac0b217d2" containerID="ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295" exitCode=0
Sep 18 19:20:59.228823 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.228709    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"
Sep 18 19:20:59.228823 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.228755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr" event={"ID":"e2e29316-03b2-4824-bd7f-db9ac0b217d2","Type":"ContainerDied","Data":"ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295"}
Sep 18 19:20:59.228963 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.228846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr" event={"ID":"e2e29316-03b2-4824-bd7f-db9ac0b217d2","Type":"ContainerDied","Data":"71310e818898e2025d154f798240a65b8d22964788d1c79fa3de645ba17179b2"}
Sep 18 19:20:59.228963 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.228900    2582 scope.go:117] "RemoveContainer" containerID="ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295"
Sep 18 19:20:59.230231 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.230203    2582 generic.go:356] "Generic (PLEG): container finished" podID="8c482324-3b11-4231-882f-c53c876eec57" containerID="6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f" exitCode=0
Sep 18 19:20:59.230345 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.230245    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"
Sep 18 19:20:59.230345 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.230271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv" event={"ID":"8c482324-3b11-4231-882f-c53c876eec57","Type":"ContainerDied","Data":"6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f"}
Sep 18 19:20:59.230345 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.230319    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv" event={"ID":"8c482324-3b11-4231-882f-c53c876eec57","Type":"ContainerDied","Data":"6b6e6675615e47e1513d86f5cd21947dd85f5feee0acb6c9d118a736b76c4d6f"}
Sep 18 19:20:59.237968 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.237869    2582 scope.go:117] "RemoveContainer" containerID="ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295"
Sep 18 19:20:59.238235 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:20:59.238217    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295\": container with ID starting with ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295 not found: ID does not exist" containerID="ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295"
Sep 18 19:20:59.238306 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.238242    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295"} err="failed to get container status \"ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295\": rpc error: code = NotFound desc = could not find container \"ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295\": container with ID starting with ea2e4fecfa93433f774d671e329137f98aec05498a161ec9bec5f92863984295 not found: ID does not exist"
Sep 18 19:20:59.238306 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.238261    2582 scope.go:117] "RemoveContainer" containerID="6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f"
Sep 18 19:20:59.246133 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.246115    2582 scope.go:117] "RemoveContainer" containerID="6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f"
Sep 18 19:20:59.246368 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:20:59.246348    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f\": container with ID starting with 6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f not found: ID does not exist" containerID="6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f"
Sep 18 19:20:59.246434 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.246373    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f"} err="failed to get container status \"6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f\": rpc error: code = NotFound desc = could not find container \"6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f\": container with ID starting with 6bedc39396b06f1c6fe88fa9d93aab1b4c31cffe8d8176032103aaaa227e758f not found: ID does not exist"
Sep 18 19:20:59.270243 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.270188    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"]
Sep 18 19:20:59.278632 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.278611    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8f9642-c46b77db6-tphpv"]
Sep 18 19:20:59.315152 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.315132    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"]
Sep 18 19:20:59.316984 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.316964    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f9642-77f9f4769c-x5rwr"]
Sep 18 19:20:59.613782 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.613713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:20:59.955503 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:20:59.955477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:00.334333 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:00.334257    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c482324-3b11-4231-882f-c53c876eec57" path="/var/lib/kubelet/pods/8c482324-3b11-4231-882f-c53c876eec57/volumes"
Sep 18 19:21:00.334672 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:00.334619    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e29316-03b2-4824-bd7f-db9ac0b217d2" path="/var/lib/kubelet/pods/e2e29316-03b2-4824-bd7f-db9ac0b217d2/volumes"
Sep 18 19:21:01.605931 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:01.605897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:02.031480 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:02.031445    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:21:03.458207 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:03.458168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:10.150997 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:10.150939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:10.436029 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:10.435938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:10.645120 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:10.645087    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:11.410032 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:11.410001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:13.452485 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:13.452451    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:14.784032 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:14.784004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:16.245687 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:16.245644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:21.680449 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:21.680415    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"]
Sep 18 19:21:21.680854 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:21.680619    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" podUID="e0f9d0cb-02fe-437f-93c2-4b2299ec2122" containerName="maas-api" containerID="cri-o://202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f" gracePeriod=30
Sep 18 19:21:21.752001 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:21.751975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:21.889026 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:21.889001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:21.923169 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:21.923148    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:21:22.011871 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.011806    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:22.027818 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.027792    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-openshift-service-ca\") pod \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") "
Sep 18 19:21:22.027934 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.027829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-metrics-certs\") pod \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") "
Sep 18 19:21:22.028004 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.027948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhlk9\" (UniqueName: \"kubernetes.io/projected/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-kube-api-access-xhlk9\") pod \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") "
Sep 18 19:21:22.028059 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.028042    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls\") pod \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\" (UID: \"e0f9d0cb-02fe-437f-93c2-4b2299ec2122\") "
Sep 18 19:21:22.028203 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.028162    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0f9d0cb-02fe-437f-93c2-4b2299ec2122" (UID: "e0f9d0cb-02fe-437f-93c2-4b2299ec2122"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:21:22.028348 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.028328    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-openshift-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:22.030206 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.030183    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e0f9d0cb-02fe-437f-93c2-4b2299ec2122" (UID: "e0f9d0cb-02fe-437f-93c2-4b2299ec2122"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:22.030302 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.030239    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0f9d0cb-02fe-437f-93c2-4b2299ec2122" (UID: "e0f9d0cb-02fe-437f-93c2-4b2299ec2122"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:22.030345 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.030324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-kube-api-access-xhlk9" (OuterVolumeSpecName: "kube-api-access-xhlk9") pod "e0f9d0cb-02fe-437f-93c2-4b2299ec2122" (UID: "e0f9d0cb-02fe-437f-93c2-4b2299ec2122"). InnerVolumeSpecName "kube-api-access-xhlk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:21:22.129415 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.129387    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-metrics-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:22.129539 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.129421    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhlk9\" (UniqueName: \"kubernetes.io/projected/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-kube-api-access-xhlk9\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:22.129539 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.129438    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f9d0cb-02fe-437f-93c2-4b2299ec2122-maas-api-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:22.194114 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.194090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:22.347514 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.347490    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0f9d0cb-02fe-437f-93c2-4b2299ec2122" containerID="202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f" exitCode=0
Sep 18 19:21:22.347619 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.347537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" event={"ID":"e0f9d0cb-02fe-437f-93c2-4b2299ec2122","Type":"ContainerDied","Data":"202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f"}
Sep 18 19:21:22.347619 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.347549    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"
Sep 18 19:21:22.347619 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.347575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" event={"ID":"e0f9d0cb-02fe-437f-93c2-4b2299ec2122","Type":"ContainerDied","Data":"b09cb8299a64843078b94c5a48e9434a9db0ceecf2424a9000c4516e512dac50"}
Sep 18 19:21:22.347619 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.347596    2582 scope.go:117] "RemoveContainer" containerID="202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f"
Sep 18 19:21:22.356990 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.356974    2582 scope.go:117] "RemoveContainer" containerID="202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f"
Sep 18 19:21:22.357252 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:21:22.357233    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f\": container with ID starting with 202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f not found: ID does not exist" containerID="202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f"
Sep 18 19:21:22.357307 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.357258    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f"} err="failed to get container status \"202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f\": rpc error: code = NotFound desc = could not find container \"202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f\": container with ID starting with 202e6e3ea61a28e6432926d241cc29614e14610a0b211aca4ad813ab0c76dd1f not found: ID does not exist"
Sep 18 19:21:22.368390 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.368358    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"]
Sep 18 19:21:22.370856 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.370833    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s"]
Sep 18 19:21:22.821486 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.821462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:22.978178 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:22.978143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:23.908271 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:23.908217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:24.335824 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:24.335739    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f9d0cb-02fe-437f-93c2-4b2299ec2122" path="/var/lib/kubelet/pods/e0f9d0cb-02fe-437f-93c2-4b2299ec2122/volumes"
Sep 18 19:21:24.741302 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:24.741268    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:21:24.876529 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:24.876484    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-feb80d-5777574b9-2484s" podUID="e0f9d0cb-02fe-437f-93c2-4b2299ec2122" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.68:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:21:30.667073 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:30.667039    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:30.870702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:30.870671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:31.088964 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:31.088848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:32.433989 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:32.433941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:33.697489 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:33.697454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:42.340725 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.340692    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"]
Sep 18 19:21:42.345682 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.341441    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz" podUID="36821d1a-405e-4a1c-b66c-b53460494109" containerName="maas-api" containerID="cri-o://6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1" gracePeriod=30
Sep 18 19:21:42.435241 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.435197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:42.598900 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.598818    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:21:42.699044 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.699014    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v84wx\" (UniqueName: \"kubernetes.io/projected/36821d1a-405e-4a1c-b66c-b53460494109-kube-api-access-v84wx\") pod \"36821d1a-405e-4a1c-b66c-b53460494109\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") "
Sep 18 19:21:42.699213 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.699060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-metrics-certs\") pod \"36821d1a-405e-4a1c-b66c-b53460494109\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") "
Sep 18 19:21:42.699213 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.699108    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36821d1a-405e-4a1c-b66c-b53460494109-openshift-service-ca\") pod \"36821d1a-405e-4a1c-b66c-b53460494109\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") "
Sep 18 19:21:42.699213 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.699170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-maas-api-tls\") pod \"36821d1a-405e-4a1c-b66c-b53460494109\" (UID: \"36821d1a-405e-4a1c-b66c-b53460494109\") "
Sep 18 19:21:42.699522 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.699489    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36821d1a-405e-4a1c-b66c-b53460494109-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36821d1a-405e-4a1c-b66c-b53460494109" (UID: "36821d1a-405e-4a1c-b66c-b53460494109"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:21:42.701372 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.701347    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36821d1a-405e-4a1c-b66c-b53460494109-kube-api-access-v84wx" (OuterVolumeSpecName: "kube-api-access-v84wx") pod "36821d1a-405e-4a1c-b66c-b53460494109" (UID: "36821d1a-405e-4a1c-b66c-b53460494109"). InnerVolumeSpecName "kube-api-access-v84wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:21:42.701483 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.701389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36821d1a-405e-4a1c-b66c-b53460494109" (UID: "36821d1a-405e-4a1c-b66c-b53460494109"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:42.701483 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.701448    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36821d1a-405e-4a1c-b66c-b53460494109" (UID: "36821d1a-405e-4a1c-b66c-b53460494109"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:21:42.703572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.703549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:42.800636 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.800604    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v84wx\" (UniqueName: \"kubernetes.io/projected/36821d1a-405e-4a1c-b66c-b53460494109-kube-api-access-v84wx\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:42.800636 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.800633    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-metrics-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:42.800799 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.800643    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36821d1a-405e-4a1c-b66c-b53460494109-openshift-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:42.800799 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.800653    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36821d1a-405e-4a1c-b66c-b53460494109-maas-api-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:21:42.863723 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:42.863653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:43.104081 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.104048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:43.378355 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.378311    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:43.449017 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.448981    2582 generic.go:356] "Generic (PLEG): container finished" podID="36821d1a-405e-4a1c-b66c-b53460494109" containerID="6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1" exitCode=0
Sep 18 19:21:43.449185 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.449020    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"
Sep 18 19:21:43.449185 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.449026    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz" event={"ID":"36821d1a-405e-4a1c-b66c-b53460494109","Type":"ContainerDied","Data":"6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1"}
Sep 18 19:21:43.449185 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.449072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz" event={"ID":"36821d1a-405e-4a1c-b66c-b53460494109","Type":"ContainerDied","Data":"b5b30d3596123837eb8f3cbec39efa99d974d2afafd66bd0908002a6059d6028"}
Sep 18 19:21:43.449185 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.449094    2582 scope.go:117] "RemoveContainer" containerID="6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1"
Sep 18 19:21:43.459225 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.459206    2582 scope.go:117] "RemoveContainer" containerID="6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1"
Sep 18 19:21:43.459518 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:21:43.459499    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1\": container with ID starting with 6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1 not found: ID does not exist" containerID="6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1"
Sep 18 19:21:43.459616 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.459530    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1"} err="failed to get container status \"6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1\": rpc error: code = NotFound desc = could not find container \"6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1\": container with ID starting with 6998fc7e298de234dcde4029d82c5277344511766160ae708a73a254e8e269f1 not found: ID does not exist"
Sep 18 19:21:43.478553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.478527    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"]
Sep 18 19:21:43.495644 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.495615    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-19944d-7bfcc895c8-xwczz"]
Sep 18 19:21:43.651775 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:43.651725    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:44.051229 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:44.051142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:44.335182 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:44.335102    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36821d1a-405e-4a1c-b66c-b53460494109" path="/var/lib/kubelet/pods/36821d1a-405e-4a1c-b66c-b53460494109/volumes"
Sep 18 19:21:45.813126 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:45.813093    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:21:53.903840 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:53.903761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:54.149296 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:54.149257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:54.471085 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:54.471048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:57.933524 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:57.933490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:21:59.333758 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:21:59.333720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:22:30.891926 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:22:30.891871    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:22:33.793259 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:22:33.793213    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:22.812294 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:23:22.812204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:23:23.069553 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:23:23.069451    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:23:24.538358 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:23:24.538315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:02.404074 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:24:02.404035    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:24:10.419241 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:24:10.419204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:24:18.945855 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:24:18.945814    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:24:22.436725 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:24:22.436688    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:34.296169 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:24:34.296135    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:24:34.847294 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:24:34.847261    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:24:41.642423 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:24:41.642388    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:25:06.051633 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:25:06.051590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:06.120899 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:25:06.120855    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:25:57.959874 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:25:57.959842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:13.694512 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:13.694470    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:26:22.405727 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:22.405638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:26:26.038569 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:26.038534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:29.283764 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:29.283722    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:32.012049 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:32.012009    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:26:34.072805 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.072765    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"]
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073126    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c482324-3b11-4231-882f-c53c876eec57" containerName="payload-pre-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073138    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c482324-3b11-4231-882f-c53c876eec57" containerName="payload-pre-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073147    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36821d1a-405e-4a1c-b66c-b53460494109" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073153    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="36821d1a-405e-4a1c-b66c-b53460494109" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073168    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f9d0cb-02fe-437f-93c2-4b2299ec2122" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073174    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f9d0cb-02fe-437f-93c2-4b2299ec2122" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073180    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65379eeb-7ccc-4990-bf7c-f1c455c518d8" containerName="payload-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073185    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65379eeb-7ccc-4990-bf7c-f1c455c518d8" containerName="payload-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073197    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd53805-7c34-4e51-b52c-5961c97868c0" containerName="payload-pre-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073203    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd53805-7c34-4e51-b52c-5961c97868c0" containerName="payload-pre-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073211    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda315c3-0224-482c-b69f-e575e80865ad" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073216    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda315c3-0224-482c-b69f-e575e80865ad" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073222    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e29316-03b2-4824-bd7f-db9ac0b217d2" containerName="payload-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073226    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e29316-03b2-4824-bd7f-db9ac0b217d2" containerName="payload-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073279    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65379eeb-7ccc-4990-bf7c-f1c455c518d8" containerName="payload-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073285    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda315c3-0224-482c-b69f-e575e80865ad" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073292    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f9d0cb-02fe-437f-93c2-4b2299ec2122" containerName="maas-api"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073298    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd53805-7c34-4e51-b52c-5961c97868c0" containerName="payload-pre-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073306    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e29316-03b2-4824-bd7f-db9ac0b217d2" containerName="payload-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073317    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c482324-3b11-4231-882f-c53c876eec57" containerName="payload-pre-processing"
Sep 18 19:26:34.073329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.073325    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="36821d1a-405e-4a1c-b66c-b53460494109" containerName="maas-api"
Sep 18 19:26:34.076180 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.076159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.079145 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.079122    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-73f386a2-kserve-self-signed-certs\""
Sep 18 19:26:34.091511 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.090850    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"]
Sep 18 19:26:34.133268 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.133238    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:34.166865 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.166839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tmp-dir\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.166975 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.166905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h278\" (UniqueName: \"kubernetes.io/projected/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-kube-api-access-2h278\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.166975 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.166931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-model-cache\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.166975 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.166949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-home\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.167085 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.167045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-dshm\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.167130 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.167115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tls-certs\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.267631 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.267600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tls-certs\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.267790 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.267637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tmp-dir\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.267790 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.267668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h278\" (UniqueName: \"kubernetes.io/projected/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-kube-api-access-2h278\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.267790 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.267693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-model-cache\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.267790 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.267711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-home\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.267790 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.267756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-dshm\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.268134 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.268109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tmp-dir\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.268227 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.268195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-model-cache\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.268227 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.268215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-home\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.270210 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.270191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-dshm\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.270492 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.270475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tls-certs\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.276524 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.276490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h278\" (UniqueName: \"kubernetes.io/projected/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-kube-api-access-2h278\") pod \"e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") " pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.387162 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.387087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:34.672349 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.672313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"]
Sep 18 19:26:34.674209 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:26:34.674171    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddeb9e41_7f7b_42aa_8c43_ca298d6e6a14.slice/crio-aa34d4424ca212438086441befae3de7e64d00694bd812c8e497c37406663b5a WatchSource:0}: Error finding container aa34d4424ca212438086441befae3de7e64d00694bd812c8e497c37406663b5a: Status 404 returned error can't find the container with id aa34d4424ca212438086441befae3de7e64d00694bd812c8e497c37406663b5a
Sep 18 19:26:34.880152 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.880105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq" event={"ID":"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14","Type":"ContainerStarted","Data":"5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823"}
Sep 18 19:26:34.880152 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.880155    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq" event={"ID":"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14","Type":"ContainerStarted","Data":"aa34d4424ca212438086441befae3de7e64d00694bd812c8e497c37406663b5a"}
Sep 18 19:26:34.911106 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:34.911056    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq" podStartSLOduration=0.911040997 podStartE2EDuration="911.040997ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:34.906036352 +0000 UTC m=+2518.698713284" watchObservedRunningTime="2026-09-18 19:26:34.911040997 +0000 UTC m=+2518.703717988"
Sep 18 19:26:43.172039 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:43.171989    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:26:43.665839 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:43.665805    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:26:44.388123 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:44.388090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:44.388549 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:44.388142    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:44.401003 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:44.400975    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:44.941401 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:44.941373    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:26:47.782560 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:26:47.782527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:08.977502 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:08.977458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:10.734323 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:10.734288    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"]
Sep 18 19:27:10.783827 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:10.783793    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:11.301072 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:11.301039    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:27:21.900416 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:21.900371    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:27:25.753685 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:25.753642    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq" podUID="ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" containerName="main" containerID="cri-o://5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823" gracePeriod=45
Sep 18 19:27:26.004532 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.004464    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:27:26.112413 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tls-certs\") pod \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") "
Sep 18 19:27:26.112572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tmp-dir\") pod \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") "
Sep 18 19:27:26.112572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-dshm\") pod \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") "
Sep 18 19:27:26.112572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112485    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-home\") pod \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") "
Sep 18 19:27:26.112572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112503    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-model-cache\") pod \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") "
Sep 18 19:27:26.112572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112542    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h278\" (UniqueName: \"kubernetes.io/projected/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-kube-api-access-2h278\") pod \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\" (UID: \"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14\") "
Sep 18 19:27:26.112854 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112638    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" (UID: "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:27:26.112854 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112796    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-home" (OuterVolumeSpecName: "home") pod "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" (UID: "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:27:26.112854 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112806    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" (UID: "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:27:26.113001 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112948    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tmp-dir\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:26.113001 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112968    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-home\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:26.113001 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.112981    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-model-cache\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:26.114869 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.114850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" (UID: "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:27:26.114964 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.114914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-dshm" (OuterVolumeSpecName: "dshm") pod "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" (UID: "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:27:26.114964 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.114927    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-kube-api-access-2h278" (OuterVolumeSpecName: "kube-api-access-2h278") pod "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" (UID: "ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14"). InnerVolumeSpecName "kube-api-access-2h278". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:27:26.130843 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.130814    2582 generic.go:356] "Generic (PLEG): container finished" podID="ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" containerID="5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823" exitCode=0
Sep 18 19:27:26.130972 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.130863    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"
Sep 18 19:27:26.130972 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.130930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq" event={"ID":"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14","Type":"ContainerDied","Data":"5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823"}
Sep 18 19:27:26.131091 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.130984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq" event={"ID":"ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14","Type":"ContainerDied","Data":"aa34d4424ca212438086441befae3de7e64d00694bd812c8e497c37406663b5a"}
Sep 18 19:27:26.131091 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.131008    2582 scope.go:117] "RemoveContainer" containerID="5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823"
Sep 18 19:27:26.139993 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.139966    2582 scope.go:117] "RemoveContainer" containerID="5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823"
Sep 18 19:27:26.140270 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:27:26.140249    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823\": container with ID starting with 5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823 not found: ID does not exist" containerID="5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823"
Sep 18 19:27:26.140329 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.140278    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823"} err="failed to get container status \"5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823\": rpc error: code = NotFound desc = could not find container \"5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823\": container with ID starting with 5553c0ac4e7a677970b30f53a237e619ee73e12ceffca7a02b579d84ba51d823 not found: ID does not exist"
Sep 18 19:27:26.213638 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.213615    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h278\" (UniqueName: \"kubernetes.io/projected/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-kube-api-access-2h278\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:26.213638 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.213637    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-tls-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:26.213774 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:26.213647    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14-dshm\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:29.082832 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:29.082792    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"]
Sep 18 19:27:29.552143 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:29.552106    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-73f386a2-kserve-5d84c899b7-964sq"]
Sep 18 19:27:30.334834 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:30.334796    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" path="/var/lib/kubelet/pods/ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14/volumes"
Sep 18 19:27:34.103903 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:34.103835    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:27:41.261239 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.261202    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:27:41.472150 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.472112    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-868977f559-qjqb2"]
Sep 18 19:27:41.472482 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.472470    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" containerName="main"
Sep 18 19:27:41.472531 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.472484    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" containerName="main"
Sep 18 19:27:41.472572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.472540    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeb9e41-7f7b-42aa-8c43-ca298d6e6a14" containerName="main"
Sep 18 19:27:41.476605 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.476582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.480002 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.479977    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f6fxk\""
Sep 18 19:27:41.480135 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.480118    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 19:27:41.480573 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.480543    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 19:27:41.480698 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.480627    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 19:27:41.480698 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.480651    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 19:27:41.518844 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.518758    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868977f559-qjqb2"]
Sep 18 19:27:41.555540 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.555510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-maas-api-tls\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.555734 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.555558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnskr\" (UniqueName: \"kubernetes.io/projected/46980f56-1707-4d6e-9bf2-c7ceaff9b988-kube-api-access-nnskr\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.555734 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.555581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46980f56-1707-4d6e-9bf2-c7ceaff9b988-openshift-service-ca\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.555734 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.555683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-metrics-certs\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.656751 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.656706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-metrics-certs\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.656991 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.656802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-maas-api-tls\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.656991 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.656840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnskr\" (UniqueName: \"kubernetes.io/projected/46980f56-1707-4d6e-9bf2-c7ceaff9b988-kube-api-access-nnskr\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.656991 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.656861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46980f56-1707-4d6e-9bf2-c7ceaff9b988-openshift-service-ca\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.657644 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.657558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46980f56-1707-4d6e-9bf2-c7ceaff9b988-openshift-service-ca\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.663031 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.659429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-metrics-certs\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.663031 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.659510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-maas-api-tls\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.669463 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.669431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnskr\" (UniqueName: \"kubernetes.io/projected/46980f56-1707-4d6e-9bf2-c7ceaff9b988-kube-api-access-nnskr\") pod \"maas-api-868977f559-qjqb2\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") " pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:41.786862 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:41.786748    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:42.080239 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:42.080137    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-868977f559-qjqb2"]
Sep 18 19:27:42.085820 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:27:42.085796    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46980f56_1707_4d6e_9bf2_c7ceaff9b988.slice/crio-691a9e69f2fd264fe35d1583cb58029ac688f28e05ca90dc575f0a32f834b5dd WatchSource:0}: Error finding container 691a9e69f2fd264fe35d1583cb58029ac688f28e05ca90dc575f0a32f834b5dd: Status 404 returned error can't find the container with id 691a9e69f2fd264fe35d1583cb58029ac688f28e05ca90dc575f0a32f834b5dd
Sep 18 19:27:42.087241 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:42.087218    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:27:42.210086 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:42.210049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2" event={"ID":"46980f56-1707-4d6e-9bf2-c7ceaff9b988","Type":"ContainerStarted","Data":"691a9e69f2fd264fe35d1583cb58029ac688f28e05ca90dc575f0a32f834b5dd"}
Sep 18 19:27:43.216945 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:43.216903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2" event={"ID":"46980f56-1707-4d6e-9bf2-c7ceaff9b988","Type":"ContainerStarted","Data":"4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc"}
Sep 18 19:27:43.217382 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:43.217013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:43.245491 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:43.245426    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2" podStartSLOduration=1.763820188 podStartE2EDuration="2.245406107s" totalImagesPullingTime="481.585919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:43.242462192 +0000 UTC m=+2587.035139123" watchObservedRunningTime="2026-09-18 19:27:43.245406107 +0000 UTC m=+2587.038083038"
Sep 18 19:27:43.817878 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:43.817834    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:27:49.228013 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.227931    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:49.238200 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.238163    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:27:49.418237 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.418195    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"]
Sep 18 19:27:49.421728 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.421708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.447497 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.447464    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"]
Sep 18 19:27:49.531382 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.531278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96dp\" (UniqueName: \"kubernetes.io/projected/96cc996a-ac59-4978-8a03-faa18fa10cce-kube-api-access-b96dp\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.531382 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.531334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cc996a-ac59-4978-8a03-faa18fa10cce-openshift-service-ca\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.531597 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.531424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96cc996a-ac59-4978-8a03-faa18fa10cce-maas-api-tls\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.531597 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.531500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc996a-ac59-4978-8a03-faa18fa10cce-metrics-certs\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.632467 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.632424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cc996a-ac59-4978-8a03-faa18fa10cce-openshift-service-ca\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.632658 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.632485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96cc996a-ac59-4978-8a03-faa18fa10cce-maas-api-tls\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.632658 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.632559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc996a-ac59-4978-8a03-faa18fa10cce-metrics-certs\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.632658 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.632596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96dp\" (UniqueName: \"kubernetes.io/projected/96cc996a-ac59-4978-8a03-faa18fa10cce-kube-api-access-b96dp\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.633279 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.633248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cc996a-ac59-4978-8a03-faa18fa10cce-openshift-service-ca\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.635260 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.635231    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96cc996a-ac59-4978-8a03-faa18fa10cce-metrics-certs\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.635393 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.635372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96cc996a-ac59-4978-8a03-faa18fa10cce-maas-api-tls\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.655342 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.655316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96dp\" (UniqueName: \"kubernetes.io/projected/96cc996a-ac59-4978-8a03-faa18fa10cce-kube-api-access-b96dp\") pod \"maas-api-66f68574b7-pl5l8\" (UID: \"96cc996a-ac59-4978-8a03-faa18fa10cce\") " pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:49.732582 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:49.732541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:50.030627 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:50.030595    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"]
Sep 18 19:27:50.031824 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:27:50.031792    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cc996a_ac59_4978_8a03_faa18fa10cce.slice/crio-a7ca7e92343f73ca7953b6a3069a707b788099a6916ef79342401bcf56b6be68 WatchSource:0}: Error finding container a7ca7e92343f73ca7953b6a3069a707b788099a6916ef79342401bcf56b6be68: Status 404 returned error can't find the container with id a7ca7e92343f73ca7953b6a3069a707b788099a6916ef79342401bcf56b6be68
Sep 18 19:27:50.253526 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:50.253489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8" event={"ID":"96cc996a-ac59-4978-8a03-faa18fa10cce","Type":"ContainerStarted","Data":"a7ca7e92343f73ca7953b6a3069a707b788099a6916ef79342401bcf56b6be68"}
Sep 18 19:27:51.260251 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:51.260207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8" event={"ID":"96cc996a-ac59-4978-8a03-faa18fa10cce","Type":"ContainerStarted","Data":"201f5ef3e459b9a87bc3f9171230deb44ca11bf0165b7fc3616c03071ad0a6f0"}
Sep 18 19:27:51.260698 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:51.260347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:51.299994 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:51.299938    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8" podStartSLOduration=1.518823322 podStartE2EDuration="2.299920152s" totalImagesPullingTime="781.09683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:51.299465416 +0000 UTC m=+2595.092142366" watchObservedRunningTime="2026-09-18 19:27:51.299920152 +0000 UTC m=+2595.092597082"
Sep 18 19:27:56.419215 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:56.419173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:27:57.270191 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.270163    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66f68574b7-pl5l8"
Sep 18 19:27:57.334577 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.334542    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868977f559-qjqb2"]
Sep 18 19:27:57.334772 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.334751    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2" podUID="46980f56-1707-4d6e-9bf2-c7ceaff9b988" containerName="maas-api" containerID="cri-o://4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc" gracePeriod=30
Sep 18 19:27:57.581537 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.581507    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:57.707972 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.707931    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnskr\" (UniqueName: \"kubernetes.io/projected/46980f56-1707-4d6e-9bf2-c7ceaff9b988-kube-api-access-nnskr\") pod \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") "
Sep 18 19:27:57.708139 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.707985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46980f56-1707-4d6e-9bf2-c7ceaff9b988-openshift-service-ca\") pod \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") "
Sep 18 19:27:57.708139 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.708063    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-maas-api-tls\") pod \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") "
Sep 18 19:27:57.708139 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.708096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-metrics-certs\") pod \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\" (UID: \"46980f56-1707-4d6e-9bf2-c7ceaff9b988\") "
Sep 18 19:27:57.708345 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.708317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46980f56-1707-4d6e-9bf2-c7ceaff9b988-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46980f56-1707-4d6e-9bf2-c7ceaff9b988" (UID: "46980f56-1707-4d6e-9bf2-c7ceaff9b988"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:27:57.710355 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.710316    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "46980f56-1707-4d6e-9bf2-c7ceaff9b988" (UID: "46980f56-1707-4d6e-9bf2-c7ceaff9b988"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:27:57.710355 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.710347    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46980f56-1707-4d6e-9bf2-c7ceaff9b988" (UID: "46980f56-1707-4d6e-9bf2-c7ceaff9b988"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:27:57.710509 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.710384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46980f56-1707-4d6e-9bf2-c7ceaff9b988-kube-api-access-nnskr" (OuterVolumeSpecName: "kube-api-access-nnskr") pod "46980f56-1707-4d6e-9bf2-c7ceaff9b988" (UID: "46980f56-1707-4d6e-9bf2-c7ceaff9b988"). InnerVolumeSpecName "kube-api-access-nnskr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:27:57.808905 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.808840    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnskr\" (UniqueName: \"kubernetes.io/projected/46980f56-1707-4d6e-9bf2-c7ceaff9b988-kube-api-access-nnskr\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:57.808905 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.808874    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46980f56-1707-4d6e-9bf2-c7ceaff9b988-openshift-service-ca\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:57.809127 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.808916    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-maas-api-tls\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:57.809127 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:57.808929    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46980f56-1707-4d6e-9bf2-c7ceaff9b988-metrics-certs\") on node \"ip-10-0-132-168.ec2.internal\" DevicePath \"\""
Sep 18 19:27:58.295745 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.295710    2582 generic.go:356] "Generic (PLEG): container finished" podID="46980f56-1707-4d6e-9bf2-c7ceaff9b988" containerID="4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc" exitCode=0
Sep 18 19:27:58.295745 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.295752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2" event={"ID":"46980f56-1707-4d6e-9bf2-c7ceaff9b988","Type":"ContainerDied","Data":"4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc"}
Sep 18 19:27:58.296016 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.295774    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2"
Sep 18 19:27:58.296016 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.295819    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-868977f559-qjqb2" event={"ID":"46980f56-1707-4d6e-9bf2-c7ceaff9b988","Type":"ContainerDied","Data":"691a9e69f2fd264fe35d1583cb58029ac688f28e05ca90dc575f0a32f834b5dd"}
Sep 18 19:27:58.296016 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.295841    2582 scope.go:117] "RemoveContainer" containerID="4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc"
Sep 18 19:27:58.305354 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.305326    2582 scope.go:117] "RemoveContainer" containerID="4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc"
Sep 18 19:27:58.305659 ip-10-0-132-168 kubenswrapper[2582]: E0918 19:27:58.305637    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc\": container with ID starting with 4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc not found: ID does not exist" containerID="4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc"
Sep 18 19:27:58.305725 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.305667    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc"} err="failed to get container status \"4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc\": rpc error: code = NotFound desc = could not find container \"4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc\": container with ID starting with 4ce4e2a2f7006fd13415c21afde617b6069aad222b0dea3618b6bff486ee15bc not found: ID does not exist"
Sep 18 19:27:58.326801 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.326760    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-868977f559-qjqb2"]
Sep 18 19:27:58.340556 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:27:58.340525    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-868977f559-qjqb2"]
Sep 18 19:28:00.335615 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:00.335580    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46980f56-1707-4d6e-9bf2-c7ceaff9b988" path="/var/lib/kubelet/pods/46980f56-1707-4d6e-9bf2-c7ceaff9b988/volumes"
Sep 18 19:28:04.733068 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:04.733027    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:28:09.638018 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:09.637978    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-5zqwn"]
Sep 18 19:28:12.629369 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:12.629333    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd68ff6d7-mjb6f_7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de/authorino/0.log"
Sep 18 19:28:20.045065 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:20.045033    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fb7c78d-dsqd4_10789836-47ae-4a4b-830c-4eb9f7d4616c/copy-manifests/0.log"
Sep 18 19:28:20.060328 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:20.060299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fb7c78d-dsqd4_10789836-47ae-4a4b-830c-4eb9f7d4616c/manager/0.log"
Sep 18 19:28:20.728059 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:20.728013    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66f68574b7-pl5l8_96cc996a-ac59-4978-8a03-faa18fa10cce/maas-api/0.log"
Sep 18 19:28:20.850164 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:20.850133    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829300-2kcf4_14f781b3-39f4-4425-89c1-cb50884edc1c/cleanup/0.log"
Sep 18 19:28:21.927938 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:21.927898    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd68ff6d7-mjb6f_7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de/authorino/0.log"
Sep 18 19:28:22.307850 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:22.307768    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dkn2b_3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd/kuadrant-console-plugin/0.log"
Sep 18 19:28:22.620278 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:22.620161    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5zqwn_660c1cf1-044e-444f-a156-607398e93597/limitador/0.log"
Sep 18 19:28:23.399790 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:23.399761    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cd84dd4ff-8zn5r_695dbc20-cce1-45f9-81c3-29e34b936f4d/kube-auth-proxy/0.log"
Sep 18 19:28:25.031464 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:25.031433    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5vvwk_d9e9860e-1121-447f-9f03-8a34a6430ea4/main/0.log"
Sep 18 19:28:25.156068 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:25.156013    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-knpz4_d120fce4-2afd-49c2-b6d0-50fe4fee252a/main/0.log"
Sep 18 19:28:31.503660 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:31.503617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:31.649392 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:31.649354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log" limit=1024
Sep 18 19:28:35.168686 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:35.168637    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrxqr_4d0f8445-381e-4d78-b730-ac9eb826b144/global-pull-secret-syncer/0.log"
Sep 18 19:28:35.276249 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:35.276217    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7xv6g_577500df-cb0a-418e-a573-688a25c63dbb/konnectivity-agent/0.log"
Sep 18 19:28:35.373173 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:35.373135    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-168.ec2.internal_aba6b42d05a06c053c050c26af669cbb/haproxy/0.log"
Sep 18 19:28:39.614513 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:39.614479    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd68ff6d7-mjb6f_7f0cd22e-f4c3-4f6c-83cf-547e2ac5e5de/authorino/0.log"
Sep 18 19:28:39.864827 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:39.864748    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-dkn2b_3e8fb66d-99ed-4edd-b3cc-65afb3fed7dd/kuadrant-console-plugin/0.log"
Sep 18 19:28:39.994507 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:39.994474    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-5zqwn_660c1cf1-044e-444f-a156-607398e93597/limitador/0.log"
Sep 18 19:28:41.706499 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:41.706422    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9957d98-a807-4cec-bd4a-10804ef340d7/alertmanager/0.log"
Sep 18 19:28:41.788998 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:41.788968    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9957d98-a807-4cec-bd4a-10804ef340d7/config-reloader/0.log"
Sep 18 19:28:41.848095 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:41.848070    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9957d98-a807-4cec-bd4a-10804ef340d7/kube-rbac-proxy-web/0.log"
Sep 18 19:28:41.908017 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:41.907992    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9957d98-a807-4cec-bd4a-10804ef340d7/kube-rbac-proxy/0.log"
Sep 18 19:28:41.959370 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:41.959308    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9957d98-a807-4cec-bd4a-10804ef340d7/kube-rbac-proxy-metric/0.log"
Sep 18 19:28:42.117822 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:42.117797    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9957d98-a807-4cec-bd4a-10804ef340d7/prom-label-proxy/0.log"
Sep 18 19:28:42.256191 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:42.256124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9957d98-a807-4cec-bd4a-10804ef340d7/init-config-reloader/0.log"
Sep 18 19:28:42.832050 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:42.832014    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf9vv_69c2029f-499c-4361-bc76-869618abd272/node-exporter/0.log"
Sep 18 19:28:42.922486 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:42.922461    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf9vv_69c2029f-499c-4361-bc76-869618abd272/kube-rbac-proxy/0.log"
Sep 18 19:28:42.979204 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:42.979171    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf9vv_69c2029f-499c-4361-bc76-869618abd272/init-textfile/0.log"
Sep 18 19:28:43.877390 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:43.877361    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-9x7ls_1a73b467-7eec-4c57-b74e-9b28b48d9bb9/prometheus-operator-admission-webhook/0.log"
Sep 18 19:28:43.933433 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:43.933404    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b8db5cb94-xx68t_39e6be68-db53-4cc8-b50c-410ee795f389/telemeter-client/0.log"
Sep 18 19:28:43.976089 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:43.976061    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b8db5cb94-xx68t_39e6be68-db53-4cc8-b50c-410ee795f389/reload/0.log"
Sep 18 19:28:44.014929 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.014903    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b8db5cb94-xx68t_39e6be68-db53-4cc8-b50c-410ee795f389/kube-rbac-proxy/0.log"
Sep 18 19:28:44.061445 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.061423    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d8fd8fcf-v69z9_c8532550-35be-4ae9-b209-81f6b19eee69/thanos-query/0.log"
Sep 18 19:28:44.123335 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.123305    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d8fd8fcf-v69z9_c8532550-35be-4ae9-b209-81f6b19eee69/kube-rbac-proxy-web/0.log"
Sep 18 19:28:44.188455 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.188433    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d8fd8fcf-v69z9_c8532550-35be-4ae9-b209-81f6b19eee69/kube-rbac-proxy/0.log"
Sep 18 19:28:44.232056 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.232039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d8fd8fcf-v69z9_c8532550-35be-4ae9-b209-81f6b19eee69/prom-label-proxy/0.log"
Sep 18 19:28:44.265133 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.265113    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d8fd8fcf-v69z9_c8532550-35be-4ae9-b209-81f6b19eee69/kube-rbac-proxy-rules/0.log"
Sep 18 19:28:44.305441 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.305417    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d8fd8fcf-v69z9_c8532550-35be-4ae9-b209-81f6b19eee69/kube-rbac-proxy-metrics/0.log"
Sep 18 19:28:44.432334 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.432307    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"]
Sep 18 19:28:44.432644 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.432633    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46980f56-1707-4d6e-9bf2-c7ceaff9b988" containerName="maas-api"
Sep 18 19:28:44.432689 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.432646    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="46980f56-1707-4d6e-9bf2-c7ceaff9b988" containerName="maas-api"
Sep 18 19:28:44.432724 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.432718    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="46980f56-1707-4d6e-9bf2-c7ceaff9b988" containerName="maas-api"
Sep 18 19:28:44.436969 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.436954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.440633 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.440582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qnj9f\"/\"default-dockercfg-9tcdj\""
Sep 18 19:28:44.441030 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.441016    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnj9f\"/\"openshift-service-ca.crt\""
Sep 18 19:28:44.442637 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.442617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnj9f\"/\"kube-root-ca.crt\""
Sep 18 19:28:44.467818 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.467786    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"]
Sep 18 19:28:44.526405 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.526379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-podres\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.526519 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.526445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nbf\" (UniqueName: \"kubernetes.io/projected/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-kube-api-access-w9nbf\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.526519 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.526480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-sys\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.526594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.526520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-lib-modules\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.526594 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.526547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-proc\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627368 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-lib-modules\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627493 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-proc\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627493 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-podres\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627493 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nbf\" (UniqueName: \"kubernetes.io/projected/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-kube-api-access-w9nbf\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627493 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-proc\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-sys\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-sys\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-lib-modules\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.627702 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.627584    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-podres\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.638383 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.638355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nbf\" (UniqueName: \"kubernetes.io/projected/11646868-c7d4-4fee-b82d-a6a3d0a08e3f-kube-api-access-w9nbf\") pod \"perf-node-gather-daemonset-fxwbz\" (UID: \"11646868-c7d4-4fee-b82d-a6a3d0a08e3f\") " pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:44.747588 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:44.747511    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:45.030362 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:45.030327    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"]
Sep 18 19:28:45.032220 ip-10-0-132-168 kubenswrapper[2582]: W0918 19:28:45.032191    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11646868_c7d4_4fee_b82d_a6a3d0a08e3f.slice/crio-444fabd189259733367b2b2c5d40a09c016b0e42e4a98415d834dc2b547b647f WatchSource:0}: Error finding container 444fabd189259733367b2b2c5d40a09c016b0e42e4a98415d834dc2b547b647f: Status 404 returned error can't find the container with id 444fabd189259733367b2b2c5d40a09c016b0e42e4a98415d834dc2b547b647f
Sep 18 19:28:45.551135 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:45.551100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz" event={"ID":"11646868-c7d4-4fee-b82d-a6a3d0a08e3f","Type":"ContainerStarted","Data":"43b3eec5d599da3ae3332267a960e5f68a80201c0267c69d8c022465dbb97994"}
Sep 18 19:28:45.551315 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:45.551152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz" event={"ID":"11646868-c7d4-4fee-b82d-a6a3d0a08e3f","Type":"ContainerStarted","Data":"444fabd189259733367b2b2c5d40a09c016b0e42e4a98415d834dc2b547b647f"}
Sep 18 19:28:45.551315 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:45.551253    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:45.600266 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:45.600223    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz" podStartSLOduration=1.600210009 podStartE2EDuration="1.600210009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:28:45.593713483 +0000 UTC m=+2649.386390443" watchObservedRunningTime="2026-09-18 19:28:45.600210009 +0000 UTC m=+2649.392886939"
Sep 18 19:28:46.880556 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:46.880526    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58c659b464-gf48h_12557472-1266-435f-bebd-a6f3726e0ee2/console/0.log"
Sep 18 19:28:49.040579 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:49.040553    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j54tt_8b41324a-3369-42ea-aa62-aa5b19ce808e/dns/0.log"
Sep 18 19:28:49.089773 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:49.089753    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j54tt_8b41324a-3369-42ea-aa62-aa5b19ce808e/kube-rbac-proxy/0.log"
Sep 18 19:28:49.294857 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:49.294792    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hf5v7_aba42289-c2a5-4752-87a8-f616b4372de0/dns-node-resolver/0.log"
Sep 18 19:28:49.996723 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:49.996693    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-78vcs_9b2eac25-004e-49a9-ba24-15fe12ebfe13/node-ca/0.log"
Sep 18 19:28:51.530582 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:51.530557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cd84dd4ff-8zn5r_695dbc20-cce1-45f9-81c3-29e34b936f4d/kube-auth-proxy/0.log"
Sep 18 19:28:51.566835 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:51.566811    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qnj9f/perf-node-gather-daemonset-fxwbz"
Sep 18 19:28:52.715919 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:52.715868    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kbzzh_2c7ee963-be99-40a0-821d-4d8fc461a0c6/serve-healthcheck-canary/0.log"
Sep 18 19:28:53.598727 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:53.598694    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2twxf_03f46265-978a-4df5-a719-87f818fe09a1/kube-rbac-proxy/0.log"
Sep 18 19:28:53.672530 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:53.672513    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2twxf_03f46265-978a-4df5-a719-87f818fe09a1/exporter/0.log"
Sep 18 19:28:53.733598 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:53.733576    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2twxf_03f46265-978a-4df5-a719-87f818fe09a1/extractor/0.log"
Sep 18 19:28:55.410059 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:55.410034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cd84dd4ff-8zn5r_695dbc20-cce1-45f9-81c3-29e34b936f4d/kube-auth-proxy/0.log"
Sep 18 19:28:57.371572 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:57.371541    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fb7c78d-dsqd4_10789836-47ae-4a4b-830c-4eb9f7d4616c/manager/0.log"
Sep 18 19:28:57.397538 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:57.397517    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fb7c78d-dsqd4_10789836-47ae-4a4b-830c-4eb9f7d4616c/copy-manifests/0.log"
Sep 18 19:28:57.542863 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:57.542834    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cd84dd4ff-8zn5r_695dbc20-cce1-45f9-81c3-29e34b936f4d/kube-auth-proxy/0.log"
Sep 18 19:28:58.743793 ip-10-0-132-168 kubenswrapper[2582]: I0918 19:28:58.743760    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-bf466d58c-7dknq_1d254738-55cf-4e18-ae85-ca1fe7f0b9c3/manager/0.log"